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ABSTRACT 


Leadership is erueial in any organization. Exeeutive leadership provides guidanee, 
direetion and strueture. First-line supervisors ensure that the vision, guidanee and 
direetion of exeeutive leaders beeome reality in an organization, and they also are 
responsible for the morale and motivation of nonsupervisory employees. Historieally, 
federal ageneies have struggled with developing and implementing seleetion proeesses 
that produee effeetive first-line supervisors. 

This researeh examines the four most important tenets of a seleetion proeess. 
These four tenets inelude laws and regulations that govern first-line supervisor seleetion 
proeesses, the aeeepted eompeteneies assoeiated with effeetive first-line supervisors, 
assessment tools used in the seleetion proeess and most effeetive measures in assessing 
organizational performanee and first-line supervisors. Using a multi-method approaeh 
that ineludes survey data, interviews and poliey review, this study eompares first-line 
supervisor seleetion proeesses for effeetive and less effeetive federal ageneies, as 
measured by the Federal Human Capital Survey (FHCS)/Federal Employee Viewpoint 
Survey (FedView). Finally, eonelusions and reeommendations are offered for developing 
standards and improving first-line supervisor seleetion proeesses in DHS and the rest of 
the federal government. 
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EXECUTIVE SUMMARY 


The foundation of any organization is its leadership. Executive leadership is crucial in 
organizations, since such leaders provide guidance, direction, and structure. However, it 
is the leaders found on the front lines, the first-line supervisors, who establish a stable 
foundation for an organization. It is these supervisors who are directly responsible for 
day-to-day leadership, coaching, mentoring, job performance and retention of 
nonsupervisory employees (Human Resources Management Panel, 2003). First-line 
supervisors ensure that the vision, guidance and direction of executive leaders become 
reality in an organization. Because of this, first-line supervisors are considered an 
important corporate leadership asset. To ensure that an organization best uses this asset, 
leaders should dedicate resources to the selection and development of first-line 
supervisors. Historically, federal agencies have struggled with developing and 
implementing first-line supervisor selection programs that produce effective supervisors 
who positively contribute to the performance of their agencies (Human Resources 
Management Panel, 2003). 

Federal agencies are required to follow policies, regulations and laws when 
selecting first-line supervisors. Title 5 Code Federal Regulations, Parts 300, 332, 335, 
and 412, specifically addresses the selection and development of first-line supervisors in 
the federal government; however, these overarching regulations are very general and 
provide no specific guidance for the selection of first-line supervisors. Ultimately, 
federal agencies are responsible for developing their own selection process to ensure the 
general guidance of Title 5 is followed. Additionally, other regulations such as 5 United 
Stated Code, Section 3321(a)(2), provide for “a period of probation...before initial 
appointment as a supervisor or manager becomes final.” This probationary period is 
meant to evaluate new supervisors to determine if they are meeting the requirements of 
the supervisor position; however, several government reports have determined that this 
probationary period in not being used as it was intended and ultimately is ineffective. 
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One report states, “The probationary period does not appear to weed out those 
supervisors who are not performing well.” (Offiee of Merit Systems Oversight and 
Effectiveness, 2001, p. 18) 

Over the past 20 years, several Office of Personnel Management (0PM) and 
Merit Service Protection Board (MSPB) reports have been published that scrutinize the 
selection process of first-line supervisors (Office of Personnel Management, 2010; Office 
of Merit Systems Oversight and Effectiveness, 2001; El.S. Merit Systems Protection 
Board, 1989; El.S. Merit Systems Protection Board, 2008; EJ.S. Merit Systems Protection 
Board, 2010). However, it does not appear that action has been taken on many of the 
recommendations from these reports. There is no consistent selection process for first- 
line supervisors used by federal agencies; however, there is a clear argument that every 
federal agency is different and therefore requires the latitude to implement their own 
selection process. Much like the probationary requirement for first-line supervisors, if 
the federal government does not establish more than passive guidance for the selection of 
first-line supervisors, then the effectiveness of first-line supervisors in the federal 
government is not likely to improve. Ultimately, the performance of each agency, and 
the federal government as a whole, will suffer. In this post-9/11 environment where 
collaboration, coordination and leadership at the frontlines is critical to the success of 
homeland security, can the Unites States afford to have ineffective first-line supervisors? 

This study compares first-line supervisor selection processes for effective and less 
effective federal agencies as measured by the Eederal Human Capital Survey 
(PHCS)/Pederal Employee Viewpoint Survey (EedView). This examination includes 
analysis of: (i) the selection processes used and accepted in federal agencies; (ii) first-line 
supervisor competencies and (iii) the relationship between different selection processes 
and first-line supervisor effectiveness in federal agencies, specifically those that impact 
homeland security. 

Key results from this study reveal several interesting findings that can provide 

insights to human resource personnel, selecting officials, agency executives and scholars. 

As expected, the results do not answer the “one” thing that makes first-line supervisors 

effective, but provides patterns and best practices for those agencies involved in this 
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study. These patterns and best practices logically point to recommendations that federal 
agencies can use to establish first-line supervisor selection processes that produce 
effective first-line supervisors. 

Findings : 

• Laws and regulations for first-line supervisor selection are very general 
and only set parameters for federal agencies. Although 0PM provides 
tools for the selection process of first-line supervisors, they fall short on 
setting specific steps or methods to follow for the selection process of 
first-line supervisors. 

• Agencies examined in this study used competencies recommended by the 
Merit Systems Protection Board for first-line supervisors. 

• Agencies used elements of the first-line supervisor selection process 
provided by 0PM, but the assessment tools used by each agency greatly 
varied. 

• When measured by the Federal Human Capital Survey (FHCS)/Federal 
Employee Viewpoint Survey (FedView), agencies that fall under DHS 
tended to have less effective first-line supervisors. 

• The agencies that had less effective first-line supervisors relied heavily on 
online administered assessment tools and self-assessment. 

• The agencies that had effective first-line supervisors used multi-method 
selection processes that contained assessments tools that required 
applicant interaction in demonstrating behaviors consistent with 
competencies being sought. 

Recommendations: 

• 0PM should assume a more active role in the selection process of first- 
line supervisors. They should require agencies to use assessment tools 
that are comprehensive in evaluating the behaviors associated with the 
competencies for first-line supervisors. Agencies should not just rely on 
self-assessments of past experience and training. 

• DHS should establish a common first-line supervisor assessment test. 
Consistent with Government Accountability Office (GAO) 
recommendations of implementing management functions and 
strengthening performance measurements, this test would be a step in 
providing more stability and consistency in these areas for agencies that 
fall under DHS. Ultimately, this test will cultivate a culture of 
connectedness and establish standards within DHS. 

• Agencies should use a multi-method approach in selecting first-line 
supervisors that contain an assessment tool that requires applicant 
interaction but avoid self-assessment. 
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• Using the First-Line Supervisor Index (FLSI) developed in this study, 
ageneies should begin to use the FedView survey results for first-line 
supervisors to traek first-line supervisor effeetiveness within their ageney. 

• By eombining and sharing common assessment tools and methods, 
agencies can lower the financial burden associated with comprehensive 
and fair first-line supervisor selection processes. 
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I. 


INTRODUCTION 


The fastest and most direct way to strengthen Federal workforce 
performance is to improve the supervision employees receive. 

Merit System Protection Board (2010) 


A. PROBLEM STATEMENT 

Leadership is crucial in any organization. Executive leadership provides 
guidance, direction, and structure. Leaders on the front lines, the first-line supervisors, 
establish a stable foundation for an organization. It is these supervisors who are directly 
responsible for day-to-day leadership, coaching, mentoring, job performance and 
retention of nonsupervisory employees (Human Resources Management Panel, 2003). 
First-line supervisors ensure that the vision, guidance and direction of executive leaders 
become reality in an organization. Because of this, first-line supervisors are considered a 
pivotal corporate leadership asset. 

Today, organizations should dedicate resources to the selection and development 
of first-line supervisors. This will ensure that organizations better use this asset; 
however, in the past, federal agencies have struggled with developing and implementing 
first-line supervisor selection programs that produce effective supervisors who positively 
contribute to the performance of their agencies (Human Resources Management Panel, 
2003; U.S. Merit Systems Protection Board, 2010). 

Historically, for most federal agencies, a first-line supervisor was not the primary 
emphasis of the agency. As part of post-9/11 environment, first-line supervisors more 
than ever need to motivate and mentor employees and encourage collaboration and 
coordination within the workplace and with stakeholders. Additionally, federal agencies 
are becoming more decentralized, relying more heavily on the decisions being made at 
the lowest level. Decentralization creates a more efficient security posture, but relies 
more heavily on the first-line supervisor. This only can occur if the selection process 
produces supervisors who have more than just technical expertise. First-line supervisors 
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need to have and be able to demonstrate the required competeneies as soon as they are 
promoted to first-line supervisor, leading in today’s dynamie environment. 

Office of Personnel Management (0PM), Merit System Protection Board (MSPB) 
and several private companies have refined, through research and case studies, the 
competencies that are most important for first-line supervisors (Human Resources 
Management Panel, 2003; Office of Merit Systems Oversight and Effectiveness, 2001; 
Office of Personnel Management, 2010; U.S. Merit Systems Protection Board, 1989; 
U.S. Merit Systems Protection Board, 2010). Additionally, this research shows a strong 
positive correlation between perceived effectiveness of supervisors and actual success of 
supervisors and their organization (Brewer, 2005; O’Driscoll & Whitehouse, 2005; 
Thompson, 2007). However, research has not been conducted that relates first-line 
supervisor selection processes employed by various federal agencies to success and 
effectiveness of an organization. 

The federal government’s selection process for first-line supervisors needs to 
identify and select first-line supervisors who are effective and add to the overall 
performance of their respective agencies. It does not appear that all first-line supervisor 
selection processes used in the federal government are accomplishing this goal. If the 
first-line supervisor selection process is not addressed, it will create a leadership void that 
is directly connected to the success and effectiveness of federal agencies. 

Over the past 20 years, several 0PM and MSPB reports have been published that 

scrutinize the selection process of first-line supervisors (Office of Personnel 

Management, 2010; Office of Merit Systems Oversight and Effectiveness, 2001; U.S. 

Merit Systems Protection Board, 2010; U.S. Merit Systems Protection Board, 1989; U.S. 

Merit Systems Protection Board, 2008). However, it does not appear that action has been 

taken on many of the recommendations from these reports. There appears to be no 

consistent selection process for first-line supervisors used by federal agencies; however, 

there is a clear argument that every federal agency is different and therefore requires the 

latitude to implement their own selection process. Much like the probationary 

requirement for first-line supervisors, if the federal government does not establish more 

than passive guidance for the selection of first-line supervisors, then the effectiveness of 
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first-line supervisors in the federal government is not likely to improve. Ultimately, the 
performanee of each agency, and the federal government as a whole, will suffer. In this 
post-9/11 environment where collaboration, coordination and leadership at the frontline 
levels is critical to the success of homeland security, the United States cannot afford to 
have ineffective first-line supervisors. 

B. FOCUS OF THIS STUDY AND RESEARCH QUESTIONS 

This thesis will use the Federal Human Capital Survey (FHCS) and Federal 
Employee Viewpoint Survey (FedView) to identify agencies that have first-line 
supervisors that are more or less effective and then examine: 1. the selection processes 
used and accepted in federal agencies; 2. first-line supervisor competencies; 3. the 
relationship between different selection processes and 4. first-line supervisor 
effectiveness in federal agencies, specifically those that impact homeland security. 

I. Research Question(s) 

Primary : How do first-line supervisor selection processes vary in effective versus 
less effective federal agencies as measured by the FHCS/FedView? 

Sub-questions: 

1. What laws or regulations govern the selection of first-line supervisors in 
government? 

2. What competencies are most often attributed to effective first-line 
supervisors? 

3. What important elements are part of a first-line supervisor selection 
process? 

4. Using the FHCS/FedView data from years 2006, 2008, 2010 and a 
composite first-line supervisor index, which agencies have the highest and 
lowest first-line supervisor effectiveness rating? 

5. What are the characteristics of the selection processes that tend to be used 
in federal agencies that have the highest and lowest supervisory ratings? 

C. SIGNIFICANCE OF THIS RESEARCH 

This study will add to the existing literature on first-line supervisors in the federal 
government by analyzing the relationship of the selection process to its effectiveness in 
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federal serviee. As the federal government continues to strive to produce a high- 
performing workforce, this study could be used to improve first-line supervisor selection. 
Lastly, as homeland security agencies and stakeholders evaluate best practices in 
improving effectiveness in homeland security related issues and priorities, this study will 
potentially provide best practices for selecting first-line supervisors in their respective 
organizations or agencies. 

D. LIMITATIONS OF THIS RESEARCH 

There are several potential shortfalls and limitations associated with this study. 
First, relying on analysis from survey data to evaluate first-line supervisor selection 
processes may not provide a comprehensive picture of the causes and effects of effective 
or ineffective selection processes. However, as mentioned above, recent studies have 
resulted in a strong positive correlation between perceived effectiveness of supervisors 
and actual success of supervisors and their organization (Brewer, 2005; O’Driscoll & 
Whitehouse, 2005; Thompson, 2007). This, coupled with interviews of employees who 
are involved in the first-line supervisor selection process, will provide valuable insight 
into the first-line supervisor selection processes in the federal agencies that are effective 
and potentially ineffective as defined by a First-Line Supervisor Index. 

In conducting interviews of employees involved in the first-line supervisor 
selection process, another potential limitation is to assume all agencies have only one 
first-line supervisor selection process for all lines of business within an agency. Because 
selection processes may vary, it is important to ensure interviews capture as many of the 
first-line supervisor selection processes for a given agency as possible. This will increase 
the validity of the interview results and their correlation to the statistical data. 

Another shortfall of this research could occur if this study was taken 
independently and not associated to past research and reports completed discussing first- 
line supervision in the federal government. This study is intended to provide additional 
information and possible recommendations in improving first-line supervisors in the 
federal government during a time where more reliance on first-line supervisors is 
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required to effieiently address the eomplexities of homeland seeurity. Therefore, this 
study should be taken as one pieee of a larger body of knowledge and researeh. 

These shortfalls and limitations are important to keep in mind; however, in order 
to impact change in an organization or process, new, innovative thinking is required. 
This study will provide new insight into how the selection process for first-line 
supervisors impacts effectiveness of first-line supervisors in federal agencies and, in turn, 
organization effectiveness. 

E. ORGANIZATION OF THIS STUDY 

This study is organized into five chapters. As indicated above. Chapter I explains 
the purpose and limitations of this study and how the topic of effective first-line 
supervisor selection processes will be discussed and addressed. Chapter II defines and 
explains the four most important tenets of a selection process in the federal government 
as defined by pertinent literature on each tenet. These four tenets include the laws and 
regulations that govern first-line supervisor selection processes, the accepted 
competencies associated with effective first-line supervisors, assessment tools used in the 
selection process and most effective measures in assessing organizational performance 
and first-line supervisors. Once the foundation of this topic is established in Chapter II, 
the research method is explained in Chapter III. A multi-method approach is used in this 
study to examine effective first-line supervisor selection processes in the federal 
government. Chapter IV then examines the results of the composite supervisor index and 
the finding from the interviews. In the last chapter, the author offers a conclusion and 
recommendations for improving first-line supervisor selection processes. 
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II. TENETS OE EIRST-LINE SUPERVISORY SELECTION 


A. OVERVIEW 

The first-line supervisor seleetion proeess in the government has four important 
tenets. Similar to the construction of an Egyptian pyramid, these tenets build on each 
other. If one is missing or unstable, the others will fail. These four tenets are described 
in this chapter: 1) the laws and regulations that govern first-line supervisor selection 
processes, 2) generally accepted competencies associated with effective first-line 
supervisors, 3) assessment methods and tools used in the selection processes and 4) 
overall measures to assess the first-line supervisor selection processes. 

B. LAWS AND REGULATIONS THAT GOVERN THE FEDERAL 

GOVERNMENT FIRST-LINE SUPERVISOR SELECTION PROCESS 

When selecting first-line supervisors, federal agencies, through their authority to 
manage personnel management functions within their agency, are required to follow 
federal policies, regulations and laws. This section describes the federal regulations that 
address supervisor selection, recruitment, merit promotion and supervisory development 
(Title 5 CFR., Parts 300, 332, 335, 412), nondiscriminatory assessment methods (Title 29 
CFR, Part 1607) and the delegation of authority for selection processes and probationary 
requirements (Title 5 USC Parts 1104 and 3321). 

I. Title 5 Code of Federal Regulations (5 C.F.R.) 
a. Part 300 

This regulation addresses basic employment practices, general 
employment information and the identification and selection of first-line supervisors. 
This regulation also defines employment practices as the “development and use of 
examinations, qualification standards, tests, and other measurement instruments” (5 

C. F.R. Part 300). It goes on to suggest that measurement instruments used should: 

1. Be practical in character and as far as possible relate to matters that fairly 
test the relative capacity and fitness of candidates for the jobs to be filled; 


7 



2. Result in selection from among the best qualified candidates; 

3. Be developed and used without discrimination because of race, color, 
religion, sex, age, national origin, partisan political affiliation or other 
nonmerit grounds; and 

4. Ensure the candidate opportunity for appeal or administrative review, as 
appropriate. (5 C.F.R. Part 300) 

Clearly, Part 300 provides parameters for selection of supervisors, but it 
falls short of implementing specific required practices which then leaves substantial grey 
area for agencies to apply these regulations as needed within their organization. 
Consequently, first-line supervisor selection processes are often inconsistent and lack 
standardization. 

b. Part 332 and 335 

These regulations provide guidelines for recruiting supervisors through 
competitive examination and through promotion and internal placement respectively. 
Part 335 describes specifics related to the merit promotion plan and merit promotion 
requirement guidelines. In Part 335(b), the regulation provides five requirements for 
agencies to follow when promoting supervisors. These requirements are: 

1. Agencies must establish procedures for promoting employees, which are 
based on merit and are available in writing to candidates. 

2. Areas of consideration must be sufficiently broad to ensure the availability 
of high quality candidates, taking into account the nature and level of the 
positions covered. 

3. To be eligible for promotion or placement, candidates must meet the 
minimum qualification standards prescribed by the Office of Personnel 
Management (0PM). Methods of evaluation for promotion and placement, 
and selection for training which leads to promotion, must be consistent 
with instructions in part 300, subpart A, of this chapter. 

4. Selection procedures will provide for management's right to select or not 
select from among a group of best qualified candidates. They will also 
provide for management's right to select from other appropriate sources, 
such as reemployment priority lists, reinstatement, transfer, handicapped, 
or Veteran Recruitment Act eligible or those within reach on an 
appropriate 0PM certificate. 
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5. Administration of the promotion system will inelude reeordkeeping and 
the provision of neeessary information to employees and the publie, 
ensuring that individuals' rights to privaey are proteeted. Eaeh ageney 
must maintain a temporary reeord of eaeh promotion suffieient to allow 
reeonstruetion of the promotion aetion, ineluding doeumentation on how 
eandidates were rated and ranked. These reeords may be destroyed after 2 
years or after the program has been formally evaluated by 0PM 
(whiehever eomes first) if the time limit for grievanee has lapsed before 
the anniversary date. (5 C.F.R. Part 335(b)) 

Although in 5 C.F.R. Part 332 and 335 the government appears to set 
parameters for ageneies to follow when seleeting first-line supervisors, in Part 335(b) 
requirement (4), the government seems to leave the ultimate diseretion for seleetion of 
first-line supervisors to the seleeting manager. This regulation then presumes that the 
“best qualified eandidates” have been properly vetted prior to seleetion by the manager; 
however, no regulation exists that provides for proper vetting of the applieants to ereate a 
pool of best qualified eandidates. This is a substantial poliey gap in the eurrent seleetion 
proeess. 


c. Part 412 

Supervisory development is addressed in 5 C.F.R. Part 412. This part 
requires ageneies to have written polieies that provide for the initial and eontinuing 
development of individuals in exeeutive, managerial, and supervisory positions, and 
eandidates for those positions. The regulation does not require pre-supervisory 
development and does not require a vetting proeess when establishing seleetion proeesses 
for first-line supervisors; however, this regulation appears to provide for “pre-seleetion” 
training and development. Therefore, as a part of the ageneies seleetion proeess, 
potential eandidates for promotion eould be vetted through development or training 
aetivities. 

2. Title 29 Code of Federal Regulations (29 C.F.R.) 
a. Part 1607 

This part provides “Uniform Guidelines on Employee Seleetion 
Proeedures” (29 C.F.R. Part 1607) speoifieally addressing the federal government’s 
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selection tests and other selection procedures. These guidelines focus primarily on 
ensuring that assessment methods are valid and are not designed in a manner that 
discriminates on the grounds of race, color, religion, sex or national origin. Although 
extremely important as a standalone guideline, 0PM has also incorporated these 
guidelines in the recommendations and guidance it issues on assessment methods and 
tools for federal employees, to include first-line supervisors (U.S. Office of Personnel 
Management, 2010). 

3. 5 United States Code (5 U.S.C.) 

a. Section 1104 (as Amended by Public Law 104-52 of1995) 

This law delegates the authority of personnel management functions 
within the federal government to the Director of 0PM and authorizes the Director of 
0PM to delegate this authority to agency heads and human resource management 
officials of each agency. In summary, 5 U.S.C. § 1104 authorizes agencies to conduct 
personnel management functions, such as first-line supervisor selection, within their 
agency. 


b. Section 3321(a)(b) 

This section requires that new supervisors serve a probationary period. If 

the new supervisor does not have a successful probationary period, then the probationer is 

placed back into a position at the same grade level from which he or she came. This 

probationary period is meant to evaluate new supervisors to determine if they are meeting 

the requirements of the supervisor position. In essence, the intent is to prevent or limit 

the Peter Principle from occurring. Laurence Peter stated that “[i]n a hierarchy, every 

employee tends to rise to his level of incompetence” (Peter & Hull, 1969). The proper use 

of the evaluations for new supervisors during the probationary period allows agencies to 

identify competence problems early on and to take corrective action; however, several 

government reports have determined that this probationary period is not being used as it 

was intended and ultimately is ineffective (Office of Merit Systems Oversight and 

Effectiveness, 2001; U.S. Merit System Protection Board, 2010). One report states, “the 

probationary period does not appear to weed out those supervisors who are not 
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performing well” (Office of Merit Systems Oversight and Effectiveness, 2001, p. 18). A 
potential reason may be the lack of valid first-line supervisor evaluations. Even if 
evaluations do occur, employees often are not held accountable if they fail. Given the 
ineffectiveness of the probationary period, the government should reexamine ways new 
supervisors can be held to the standards required for supervisory positions in the federal 
government. 

4. Conclusion 

These overarching regulations, laws and guidelines are very general and provide 
agencies with specific nondiscriminatory practices that must be used when hiring or 
promoting federal employees; however, it seems specific guidance and standards for the 
selection of first-line supervisors is not addressed in a fashion that ensures clear 
understanding of what “best qualified” means in the selection process. Hence, the 
selection can be flawed. Additionally, other regulations such as 5 U.S.C., § 3321(a)(2), 
provide for “a period of probation.. .before initial appointment as a supervisor or manager 
becomes final.” Although this probationary period is meant to evaluate new supervisors 
and to determine if they are meeting the requirements of the supervisory position, it does 
not appear to be used by agencies consistently or effectively (5 U.S.C., § 3321(a)(2)). 
Ultimately, federal agencies are responsible for developing their own selection process 
within the general guidelines of Title 5 CER and 5 USC and the selection process for 
first-line supervisors in each agency is likely to be very different. This variability in 
selection procedures is also likely to lead to wide ranges of supervisory effectiveness. 

C. FIRST-LINE SUPERVISORY COMPETENCIES IN THE FEDERAL 

GOVERNMENT 

Competencies are “a set of behaviors that encompasses skills, knowledge, 
abilities, and personal attributes that, taken together, are critical to successful work 
accomplishment” (Answers.com, 2011). There are several opinions as to what 
competencies are needed to be an effective first-line supervisor. The U.S. Merit System 
Protection Board (2010) recommends 10 foundational competencies for first-line 
supervisors. These competencies are; 
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• Accountability 

• Customer service 

• Decisiveness 

• Flexibility 

• Integrity/honesty 

• Interpersonal skills 

• Oral communieation 

• Problem solving 

• Resilience 

• Written eommunieation (U.S. Merit System Proteetion Board, 2010) 

Additionally, the Merit Board (2010) reeommends that first-line supervisors 
should posses or have the capability to possess the following additional four 
competencies: 

• Human capital management 

• Developing others 

• Conflict management 

• Teambuilding 

The literature shows that it would be impossible to establish a comprehensive list 
of all competencies needed for all first-line supervisors because each organization (or 
component of an organization) will have its own specific competencies that may be 
unique to itself, or at least not applicable, to all organizations or components. Therefore, 
in addition to the above common competencies, each organization or component should 
conduct a job task analysis to identify additional competencies for eaeh first-line 
supervisor position (U.S. Merit Systems Protection Board, 1989, p. 2; U.S. Merit System 
Proteetion Board, 2010). Job task analysis is the proeess of deseribing and recording task 
specific aspects of jobs and specifying the skills and other requirements neeessary to 
perform the job. 
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D. ASSESSMENT METHODS USED BY THE FEDERAL GOVERNMENT 

The federal government uses a variety of assessment tools for supervisor selection 
and promotion. This section describes U.S. Office of Personnel Management’s 
recommendations for assessment. 

0PM, whose tagline is “recruiting, retaining and honoring a world-class 
workforce to serve the American people,” (2011) serves as a human capital advisor to 
federal agencies. As such, they provide federal agencies with human resource 
management best practices and recommendations and they are generally not responsible 
for setting rules, regulations or laws for federal agencies to follow. The selection process 
recommendations can be found on OPM’s Personnel Assessment and Selection Resource 
Center (U.S. Office of Personnel Management, 2010). Included at the Resource Center is 
an automated assessment decision tool (ADT) that links best practice assessment tools 
from the Department of Labor. 

0PM has consolidated their recommendations and best practices into a handbook 
called Delegated Examining Operations Handbook: A Guide for Federal Agency 
Examining Offices (2011). Chapter II of this handbook provides recommendations on 
selection and assessment methods for first-line supervisors including validity descriptions 
for each method, but 0PM clearly states that each process will be different depending on 
the time frame, number of applicants and desired outcomes (U.S. Office of Personnel 
Management, 2007). 0PM recommends that each agency examine these factors when 
deciding on the assessment tools used to select federal employees, including supervisors. 
Additionally, agencies are encouraged to use the OPM’s automated ADT. This tool asks 
questions regarding the expected outcomes of the selection process to include desired 
competencies, number of expected applicants and expected timeframe for completion of 
the selection process. The ADT then provides recommendations and options comparing 
time required for each assessment tool, the expertise needed for each tool and the validity 
of each tool. Table 1 contains a list of assessment tools and a comparison of the validity 
and cost of each test. 
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Table 1. Example of Assessment Decision Tool (ADT) Results (From 0PM, 2011) 


Assessment Method 

Validity! 

Face Validity/ 
Applicant 

Reactions^ 

Subgroup 

Differences^ 

Development 

Costs 

Administration 

Costs 

Accomplishment 

Records 

High 

Moderate 

Low 

Moderate 

Moderate 

Assessment Centers 

Moderate 

High 

Low 

High 

High 

Biographical Data 

(Biodata) Tests 

Moderate 

Moderate 

Moderate 

High 

Low 

Cognitive Ability Tests 

High 

Moderate 

High 

Moderate 

Low 

Emotional Intelligence 
Tests 

Moderate 

Moderate 

Low 

High 

Low 

Integrity/Honesty Tests 

Moderate 

Moderate 

Low 

High 

Low 

Job Knowledge Tests 

High 

High 

High 

High 

Low 

Personality Tests 

Moderate 

Moderate 

Low 

High 

Low 

Reference Checking 

Moderate 

Moderate 

Low 

Moderate 

Low 

Situational Judgment 

Tests 

Moderate 

High 

Moderate 

High 

Low 

Structured Interviews 

High 

High 

Low 

Moderate 

Moderate 

Training and Experience 
(T & E) Evaluations 

Low 

Moderate 

Low 

Low 

Low 

Work Samples and 
Simulations 

High 

High 

Low 

High 

High 


The multiple assessment tools mentioned above can be summarized in six general 


categories: 

1. Past Performance/Experience Evaluations 

a. Accomplishment Records 

b. Biographical Data Tests 

c. Reference Checking 

2. Training and Experience Evaluations 

a. Practical Application Evaluations 

b. Situational Judgment Tests 

c. Work Sample/Simulations 


1 Validity is the extent to which assessment scores are related to current or future job performance. 

2 Face validity is the applicant’s perception of the validity of the tool based on visual inspection. 

3 Subgroup differences refers to the level at which the assessment tool is consistent regardless of race, 
sex or origin. 
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3. Written/Standardized tests 

a. Cognitive Ability Tests 

b. Emotional Intelligence Tests 

c. Personality Tests 

4. Job Knowledge Tests 

5. Interviews 

6. Assessment Centers (0PM, 2011) 

Some of the assessments methods recommended by 0PM do not apply or are less 
applicable to supervisor selection since in most cases first-line supervisors in the federal 
government are already federal employees. For example, under the Americans with 
Disabilities Act, personality tests that meet the definition of medical examinations cannot 
be required after an offer for employment has been made. Therefore, personality tests 
could not be given to potential first-line supervisors unless they are new-hires in the 
federal government. 

Although 0PM provides multiple sources, resources and publications that 
recommend best selection and assessment practices, individual agencies ultimately have 
the discretion to choose their assessment tools. This leaves open the possibility or 
likelihood that an agency will choose an assessment tool or selection method that will 
meet the minimum federal regulations and guidelines but will fall short in choosing the 
assessment tools and methods that would have the best chance of providing each agency 
with the best qualified candidate. Perhaps this vulnerability in the selection process for 
federal employees, specifically first-line supervisors, is an important factor contributing 
to concern of first-line supervisor effectiveness that has been documented in several 
0PM, MSPB and independent organization studies (Brewer, 2005; Human Resources 
Management Panel, 2003; O’Driscoll & Whitehouse, 2005; Office of Merit Systems 
Oversight and Effectiveness, 2001). 

E. MEASURING FIRST-LINE SUPERVISORY SELECTION PROCESSES 

In order for an organization or a process within an organization to work properly. 


there must be a way of measuring its effectiveness and performance. Without this last 

15 



tenet in the seleetion of first-line supervisors, ageneies within the federal government 
eannot determine if the steps and proeedures outlined in their first-line supervisor 
seleetion proeess aetually meet the desired individual and organization outeomes. 

An examination of eurrent literature for government and private organizations 
reveals little evidence of tangible measures used to gauge the effectiveness of first-line 
supervisor selection processes. In private sector organizations, the selection processes 
tend to be more holistic, viewing the selection and development of first-line supervisors 
as one integrated process (Charan, 2008). Because of this, for private companies, current 
measures of effectiveness become their ability to fill first-line supervisor positions with 
qualified employees who have been developed and assessed over a period of time. The 
ultimate objective measurement of effectiveness for private organizations, of course, is its 
ability to remain competitive and survive in a free market economy. Unlike the private 
sector, the federal government is not concerned about profits and loss and worries less 
about survivability. Therefore, measuring effectiveness in this manner becomes much 
more difficult. 

Currently, it does not appear that federal agencies measure the effectiveness of 
their first-line supervisor selection processes in any rigorous manner. A few studies 
address perceived best practices or describe agencies going above and beyond what is 
required in Title 5 Code of Federal Regulations (CFR)'^ (Human Resources Management 
Panel, 2003; Office of Merit Systems Oversight and Effectiveness, 2001), but they do not 
discuss a rigorous method of measuring the effectiveness of first-line supervisor selection 
processes. Additionally, 0PM and MSPB appear to use extensive survey data in 
assessing organizational performance at an agency level (U.S. Merit System Protection 
Board, 2010; U.S. Office of Personnel Management, 2008), but this data does not appear 
to be used to measure more micro-level processes like first-line supervisor selection. 
However, the fact that MSPB and OMP uses perceptive measures to evaluate agency 
performance and the extensive studies connecting perceptive measures to actual 
performance (Bommer, Johnson, Rich, Podsakoff, & MacKenzie, 1995; Brewer, 2005; 

^ Title 5 CFR, Part 300, 332, 335, 412 addresses the selection and development of first level 
supervisors in the federal government. 
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Delaney & Huselid, 1996; Dess & Robinson, 1984; O’Driseoll & Whitehouse, 2005) 
indieate that in the absenee of external objeetive data, the best alternative way to evaluate 
first-line supervisor seleetion proeesses is by using internal pereeptive measures. In order 
for the seleetion proeess of first-line supervisors to be most effeetive, however, the 
federal government must develop a measurement tool to evaluate their seleetion 
proeesses. 

F. CONCLUSION 

Examination of the four tenets of first-line supervisor seleetion proeesses makes it 
elear that in order to develop and maintain an effeetive seleetion proeess, several faetors 
need to be taken into aeeount and eonstantly reevaluated to ensure the proeess is meeting 
the desired outeomes. The federal government, through laws and regulations and OEM’s 
guidanee, has established a framework that helps ageneies address three of the four tenets 
of first-line supervisor seleetion proeesses; however, the fourth tenet seems to be missing, 
possibly due to the eomplexity and uneertainty of effeetive measurement methods or 
beeause limited miero level external and internal performanee data exists. The fourth 
tenet—measuring performanee and organizational effeetiveness at an aetionable level— 
needs attention and substanee. There is no doubt that this area of study is very eomplex 
and diffieult to define. However, this diffieulty should not preelude or in any way 
dissuade seholarly debate on best praetiees and applieation of the most effeetive tools to 
assess overall organizational performanee and that of the first-line supervisor seleetion 
proeesses. Without all four tenets, first-line supervisor seleetion in the federal 
government is likely to be less effeetive. 
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III. RESEARCH METHOD 


A. OVERVIEW 

Little research has been conducted about first-line supervisor selection processes 
in the federal government. Existing studies tend to report aggregate data, with analysis 
remaining at a government-wide, or agency level. Existing studies have not 
demonstrated how the selection process of first-line supervisors correlates with overall 
agency effectiveness or first-line supervisor effectiveness. Additionally, existing studies 
do not appear to offer insights or solutions for improving the selection of first-line 
supervisors in the federal government. 

This study is a multi-method examination of the selection processes for first-line 
supervisors in effective and less effective federal government agencies. Survey data used 
in this study comes from the Eederal Human Capital Survey (EHCS) and Eederal 
Employee Viewpoint Survey (EedView). Interview data used in this study comes from 
in-depth interviews and policy reviews about selection processes in selected agencies. 
Research questions addressed by this study are listed below. 

1. Research Questions 

a. Primary Research Question 

How do first-line supervisor selection processes vary in agencies with 
effective verses less effective first-line supervisors? Eor each of the sub-questions below, 
differences between agencies with effective first-line supervisors and those with less 
effective first-line supervisors will be examined. 

b. Sub-Questions 

1. What laws or regulations support or hinder the selection of first-line 
supervisors? 

2. What competencies are most often attributed to effective first-line 
supervisors? 

3. What important elements are part of the first-line supervisor selection 
process? 
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4. 


Using the FHCS/FedView data from years 2006, 2008, 2010 and a 
eomposite first-line supervisor index, how effeetive are the first-line 
supervisors eaeh agency and which agencies have the highest and lowest 
first-line supervisor effectiveness rating? 

5. What attributes of first-line supervisor selection processes are similar or 
different in agencies with effective or less effective first-line supervisors. 

Analysis of data expected to answer each of these sub-questions provides 
a solid foundation in which to answer the main research question for this thesis. In 
essence, to fully understand a problem and be able to provide potential solutions, one 
must first dissect and analyze the parts of the problem. In doing this, the researcher will 
be able to suggest best practices for first-line supervisor selection processes in the federal 
government. Office of Personnel Management (0PM) and Merit Systems Protection 
Board (MSPB) research and reports express concern within the government about the 
effectiveness of first-line supervisors (Human Resources Management Panel, 2003; 
Office of Merit Systems Oversight and Effectiveness, 2001; U.S. Merit Systems 
Protection Board, 1989). This research will provide additional information on one key 
aspect of first-line supervisor effectiveness, the selection process. 

It is important to note that this research is not intended to recommend a 
best way to select first-line supervisors in the federal government. Several competing 
factors influence and alter the effective selection of first-line supervisors beyond what is 
discussed in this study; however, it is expected that this research will reveal insights into 
how the selection process impacts success of first-line supervisors in the federal 
government. 

B. SURVEY DATA: THE FEDERAL HUMAN CAPITAL SURVEY AND 

FEDERAL VIEW POINT SURVEY 

In 2002, 0PM initiated and instituted the first Federal Human Capital Survey 
(FHCS). Since 2002, 0PM administers this survey every two years to all federal 
employees. In 2010, 0PM changed the name of this survey to Federal Employee 
Viewpoint Survey (EedView). The EedView 2010 is an 89-item survey that includes 11 
demographic questions and 78 items measuring federal employees’ perceptions about 
how effectively the workforce is managed. Comparing the 2010 to the 2008 and 2006 
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FHCS, several differenees were noted. The 2008 FHCS had 58 items in eommon with 
the 2010 FedView and the 2006 FHCS had 57 items in eommon with the 2010 FedView. 
When eomparing the results for eaeh year, it is important to note that some questions in 
the overall survey and in the demographie questions have ehanged or were modified 
slightly in 2010 so the data for some questions needs to be reeoded to allow eomparison. 
For example, one demographie question asked all three years is: what is your 
supervisory status (U.S. Offiee of Personnel Managment, 2006)? The response options 
offered in 2010 were A. Nonsupervisor/Team leader B. Supervisor and C. 
Manager/Exeeutive. By eomparison, in 2008 and 2006, the response options offered 
were A. Nonsupervisor, B. Team Leader, C. Supervisor, D. Manager, and E. Exeeutive. 
To be eompared, responses for 2008 and 2006 must be reeoded (A and B eollapsed to A, 
and D and E eollapsed to C) to mateh options offered for 2010. 

Unlike with the EHCS, the 2010 EedView survey grouped the 89 questions into 
eight topie areas that respondents see as they take the survey. These eight topie areas are: 
personal work experienee, work unit, ageney, supervisor/team leader, leadership, 
satisfaetion, work/life, and demographies. In essenee, by ereating eight topie areas, 0PM 
has ereated indexes that ean be used to analyze the respondent’s responses. Eor example, 
the supervisor/team leader topie area eontained 11 questions that speoifieally pertain to 
the pereeptions of supervisors and team leaders in eaeh ageney.^ These eight topie areas 
or indexes are different and should not be eonfused with Human Capital Assessment and 
Aeeountability Eramework (HCAAE) indiees. 

The HCAAE indiees provide eonsistent metries for measuring progress toward 
HCAAE objeetives. The four indiees provided are: the Leadership and Knowledge 
Management Index, the Results-Oriented Performanee Culture Index, the Talent 
Management Index and the Job Satisfaetion Index. These indiees are used to assess 
government-wide performanee and trends; however, due to the more general eategories, 
the HCAAE indiees are hard to use for subsets or speeifie aspeets of one HCAAE index. 


^ It is important to note that OPM defines team leader and supervisor as those who supervise or 
oversee nonsupervisor employees. 
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Therefore, in order to examine first-line supervisor performance in a particular agency, it 
is more pertinent to use the supervisor/team leader topic area since these questions are 
more focused on the effectiveness of first-line supervisors. 

The 2006 and 2008 FHCS and 2010 FedView were self-administered Web 
surveys with paper version being sent out to agencies that did not have Web access. The 
samples for the FHCS and FedView were consistent, based on the number of respondents 
and the return rate for years 2006, 2008, and 2010. The surveys all were directed to full¬ 
time permanent employees and administered as a census to participating agencies. The 
participating agencies made up 97 percent of the executive branch workforce. For all 
three years, the response rate was over 50 percent. Table 2 shows the response rates per 
year. 


Table 2. FHCS/FedView Response Rates (After 0PM, 2006, 2008, 2010) 


Year 

Invited 

Respondents 

Response Rate 

2006 

390,657 

221,479 

57% 

2008 

417,128 

212,223 

51% 

2010 

504,609 

263,475 

52% 


According to 0PM, the purpose of FHCS/FedView is to collect data on the 
perceptions of federal employees regarding how effective agencies are managing their 
workforce; however, 0PM does not provide information that connects perceptions of 
employees to actual agency effectiveness. This could be for several reasons, one of 
which is that it is very hard to determine government agency effectiveness with a metric 
that can be applied to all agencies. 

C. AGENCY SELECTION 

Eleven agencies were selected for this study. The selection of these agencies was 
based upon similarity of the agencies’ primary or secondary missions and their 
relationship to homeland security. The following agencies are included in this study. 

• Bureau of Alcohol, Tobacco, Firearms and Explosives (ATE) 

• Drug Enforcement Administration (DEA) 

• Eederal Bureau of Investigation (EBI) 
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• Federal Emergency Management Agency (FEMA) 

• Transportation Security Administration (TSA) 

• Citizenship and Immigration Services (CIS) 

• Coast Guard (CG) 

• Customs and Border Protection (CBP) 

• Immigration and Customs Enforcement (ICE) 

• U.S. Marshals Service (USMS) 

• Secret Service (SS) 

D. DEVELOPMENT OF SUPERVISORY INDEX 

I. First-Line Supervisor Index (FLSI) 

In 2010, 0PM modified FHCS in several ways. The result of these modifications 
can be seen in the new version of this survey called the FedView. In addition to adding 
more questions to the survey, 0PM also grouped the new questions and the old questions 
that are directed at a specific topic. As mentioned above, one of these groups is called 
“My Supervisor/Team Eeader.” Eleven questions are captured in this group. Of the 
eleven questions, eight of these questions are either identical or so similar as to seem to 
invoke the same response from the respondent in the FHCS 2006 and 2008.^ 

The First-Fine Supervisor Index (FESI) used in this study contains the eight 
questions listed in Table 3. These questions were used consistently in the 0PM surveys 
administered in 2006, 2008, and 2010. To narrow the study to focus specifically on the 
perceptions of nonsupervisor employees of their first-line supervisors, respondents were 
narrowed to only include nonsupervisor. The following questions represent the FESI 
discussed in this study. 


^ Normalization of Data-During the research and analysis of the data for the three years included in 
this study, it was discovered that in several instances the data collected was not coded the same for all 
years. The researcher normalized the data, so that accurate and complete analysis could be completed. 
Those questions that applied to this study and had to be recoded and normalized can be found in Appendix 
A. 
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Table 3. First-Line Supervisor Index 


First-Line Supervisor index 


Questions 

Factor 

My supervisor supports my need to balance work and other life issues. 

B 

My supervisor/team leader provides me with opportunities to demonstrate my leadership skills. 

B 

Discussions with my supervisor/team leader about my performance are worthwhile. 

B 

My supervisor/team leader is committed to a workforce representative of all segments of society. 

B 

My supervisor/team leader provides me with constructive suggestions to improve my job performance. 

B 

Supervisors/team leaders in my work unit support employee development. 

B 

I have trust and confidence in my supervisor. 

0 

Overall, how good a job do you feel is being done by your immediate supervisor/team leader? 

0 

B=Behavior Factor 0=Outcome Factor 


2. Behavior Factor and Outcome Factor 

In developing this study, and to understand the possible relationship between 
questions within the FLSI, the researeher determined that within the eight questions of 
the FLSI, two subgroups eould be identified. These subgroups are the behavior faetor 
and outeome faetor. Although one would assume that the behaviors and outeomes of 
supervisors would be elosely assoeiated, this assumption needed to be tested to determine 
if these two faetors were indeed statistieally eorrelated. If pereeived behaviors of 
supervisors have a strong eorrelation to the pereeived outeomes within an ageney, then 
the overall FLSI has more statistieal relevanee to overall pereeived effeetiveness of first- 
line supervisors in eaeh federal ageney examined. Additionally, this in-depth validation 
of the FLSI was important sinee this index was used in determining how to eonduet the 
seeond part of the method in this study. After examining the FLSI questions, the 
researeher found that six of the eight questions seemed to be asking about the pereeived 
behaviors of supervisors or team leaders. These questions make up the behavior faetor. 
Two of the eight questions asked about the pereeived outeomes of supervisors or team 
leaders. These questions make up the outeome faetor (see Table 3). 

3. Analysis 

The FLSI was applied to all federal ageneies ineluded in this study. The analysis 
foeuses on the nonsupervisor responses to these index questions. Using analysis of 
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variance (ANOVA) and Pearson’s eorrelation statistieal analysis, analysis ean be focused 
on the pereeived effeetiveness of first-line supervisors. ANOVA and Pearson’s 
eorrelation statistieal tools are used here beeause past studies “have shown that measure 
of perceived organizational performance correlate positively with moderate to strong 
assoeiations with objeetive measures of organizational performanee” (Brewer, 2005, p. 
511). This information is presumed to identify which agencies have the most effective 
and the least effeetive first-line supervisors. Using an aggregate mean of all eight 
questions in the FLSI, the ageneies were ranked from most effeetive to least effective. 

E. INTERVIEWS 

1. Data Collection 

Based on the FLSI analysis, interviews or poliey reviews were eondueted with 
three ageneies from the group with the high performing first-line supervisors and three 
ageneies were seleeted from the group with the lowest performing first-line supervisors. 
The purpose of the interviews and poliey reviews was to gather data about the first-line 
supervisor seleetion proeess for the seleeted ageneies. If polieies or proeedures were not 
available, in-depth interviews were eondueted with agency employees who are familiar 
with the ageney’s first-line supervisor seleetion proeess. Generally, these employees 
were human resource speeialists direetly involved in the first-line supervisor selection 
process in eaeh ageney. A matrix of assessment method eharaeteristies was used during 
the interview proeess or poliey review to eapture the types of assessment tools and 
assessment faetors used by eaeh agency in the sample (see Table 4). Additionally, 
verbatim notes were taken on information provided in the interviews to eapture all 
relevant information about the first-line supervisor seleetion proeess. 

2. Interview Questions 

When a poliey review was not possible for an agency, the following questions 
were used during the interview of an employee familiar with the first-line supervisor 
selection process for the respeetive ageney. The interviews were eondueted on the 
telephone. 
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1. Describe the first-line supervisor selection process in your organization. 

2. Has the first-line supervisor selection process been changed or altered in 
the last five years? If so, can you describe the alternations that have taken 
place? 

3. Is the selection process you have described consistent throughout your 
organization? 

4. What aspect(s) of the first-line supervisor selection process used by your 
organization are most important and least important? Why? 

5. What modifications would you make to the current first-line supervisor 
selection process used in your organization? Why? 

3. Selection Process Matrix 

Inherent in the recommendations and best practices provided by 0PM, are 
specific criteria for assessing selection processes. The most important criteria for this 
study are listed in the Table 4 and used to capture important characteristics of the first- 
line supervisor selection process in each agency in this study sample. These data make 
possible the comparison and analysis of first-line supervisor selection processes and 
identification of similarities or differences between agencies with effective and those 
with less effective first-line supervisors. 


Table 4. First-Line Supervisor Assessment Tool Matrix 


First-Line Supervisor Selection Assessment Tools 


Assessment Tool 

Agency 

1 

Agency 

2 

Agency 

3 

Agency 

4 

Agency 

5 

Agency 

6 

Pre-selection Vetting 







Number of Hurdles in Selection 

Process* 







Self-Assessment Portion (Y/N) 







Past Performance Evaluations 








Accomplishment Record 







Training & Experience 







Practical Application Evaluations 








Situational Judgment Tests 







Work Sample/Simulations 







Writing Skills 







Job Knowledge Tests 
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First-Line Supervisor Selection Assessment Tools 

Assessment Tool 

Agency 

1 

Agency 

2 

Agency 

3 

Agency 

4 

Agency 

5 

Agency 

6 

Interviews 







*See Appendix B 

K 

ey 

E=EI( 

Bctronic 

AC=Assessment 

Center 

l=ln-Person 

T= Telephonic 


4. Analysis 

The focus of analysis in this study is specifically on the elements and mechanics 
of the first-line supervisor selection process. The assessment tools and selection process 
used are examined to determine similarities and differences. This includes examining the 
numbers of hurdles in a selection process, if the assessment tools and process are 
administered electronically, telephonically or in-person, and if there is a self-assessment 
portion in the process. This information, coupled with the FLSI findings, is analyzed to 
determine if there is any relationship between the specific selection process and first-line 
supervisor effectiveness rating. 

F. CONCLUSION 

Studies show that effective first-line supervisors are lacking in the federal 
government which is problematic because high performing supervisors are critical to an 
organization’s success and effectiveness (Brewer G. A., 2005; Office of Merit Systems 
Oversight and Effectiveness, 2001; Human Resources Management Panel, 2003; U.S. 
Merit Systems Protection Board, 1989). Using quantitative and qualitative approaches to 
this study, important insights about which characteristics of first-line supervisor selection 
processes are likely to produce effective first-line supervisors. Contrary to existing 
studies on first-line supervisors’ effectiveness that focus at the agency level and do not 
consider the nature of the selection process, this multi-method study provides a starting 
point for additional research focused on this important factor—the selection process. 
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IV. RESULTS 


Don’t rate potential over performanee. 


Jim Fassel 


A. INTRODUCTION 

The results of this study reveal several interesting findings that ean provide 
insights to human resouree personnel, seleeting offieials, ageney exeeutives and seholars. 
As expeeted, the results do not answer the “one” thing that makes first-line supervisors 
effective but provides patterns and best practices for those agencies involved in this 
study. The following results emerged in response to the research questions (proposed in 
Chapter III). 

B. HOW LAWS OR REGULATIONS GOVERN THE SELECTION OF 

FIRST-LINE SUPERVISORS IN GOVERNMENT 

While extensive laws and regulations provide guidance and prohibit certain 
practices for the selection of first-line supervisors, these laws and regulations tend to be 
very general and only set parameters for federal agencies. The Office of Personnel 
Management (0PM) is the federal agency responsible for providing specific guidance 
and best practices based on established laws and regulations for all human resource issues 
in the federal government. 0PM provides tools for the selection process of first-line 
supervisors, but it falls short on setting specific steps or methods for processes which 
comply with the established laws and regulations. It appears that it is up to the individual 
agencies to determine what assessment methods and steps to use when selecting first-line 
supervisors and that they are doing so without definitive oversight regarding compliance 
with laws and regulations. This was confirmed during the interviews conducted in this 
study. Each person interviewed understood that there were laws and regulations that 
governed the selection process, but few actually knew the substance of the laws and 
regulations. For the most part, those interviewed referenced 0PM as the organization 
responsible for ensuring agencies stayed within the parameters set by the laws and 
regulations; however, little evidence from the interviews explained how 0PM 
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accomplishes this regulatory mission. This could be due to the fact that the individuals 
interviewed do not have training or experience with how 0PM influences their agency. 
While this is interesting to note, ageney compliance with laws and regulations is beyond 
the seope of this study. 

C. COMPETENCIES THAT ARE MOST OFTEN ATTRIBUTED TO 

EFFECTIVE FIRST-LINE SUPERVISORS 

Competencies attributed to hrst-line supervisors were explored through interviews 
with ‘savvy insiders’ and ageney policy reviews. In general, ah agencies ineluded in this 
study used at least some of the competeneies provided by the Merit Systems Protection 
Board (MSPB), but the number of competencies varied widely by ageney and speeialty. 
Aeross ah six agencies and career speeialties examined in this study, the following 
competencies were consistently sought for first-line supervisors: 

• Aceountability 

• Flexibility 

• Integrity/honesty 

• Interpersonal skills 

• Oral communieation 

• Problem solving 

• Written eommunication 

It appears the guidance provided by 0PM and MSPB with regards to the 
competencies that first-line supervisors should possess is applied consistently across 
agencies in this sample. Furthermore, consistent with OPM’s guidance, it seems some 
ageneies cater to the speeifie needs of their first-line supervisor jobs ineorporating 
additional eompetencies that go beyond the fundamental competeneies reeommended. 
Even so, the manner in whieh agencies assess these competeneies for first-line 
supervisors seem to be inconsistent. 
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D. THE IMPORTANT ELEMENTS OF A FIRST-LINE SUPERVISOR 

SELECTION PROCESS 

All of the agencies examined in this study used elements of the first-line 
supervisor selection process provided by 0PM (outlined in Chapter II); however, the 
number of hurdles and the assessment tools used by each agency and career specialty 
greatly varied. For the most part, non-law enforcement selection processes, compared to 
law enforcement selection processes, for example, tended to use fewer assessment tools 
and the process seemed to be more streamlined and tended to be completed at the field 
office level. Additionally, the non-law enforcement career specialties all had a structured 
panel interview as part of the last hurdle in the selection process, whereas law 
enforcement career specialties did not have this same requirement (see Appendix B). The 
differences in selection processes for career specialties, specifically the number of 
hurdles and assessment tools used, could be a reflection of how each agency views the 
level of complexity for the different career specialty. However, the data collected within 
this study does not provide enough information to provide definitive information as to the 
reasons for the differences in the selection processes for the different career specialties. 

E. AGENCIES WITH THE HIGHEST AND LOWEST FIRST-LINE 

SUPERVISOR EFFECTIVENESS RATING 

The First-Line Supervisor Index (FLSI) discussed in Chapter III is a composite 
measure of eight questions that offer insight into the perceptions of nonsupervisor 
employees about the effectiveness of their first-line supervisors. Using the mean scores 
of the FLSI, agencies can be ranked and an analysis of variance (ANOVA)^ 
accomplished to determine if these mean scores demonstrate real differences (see 
Appendix C). Table 5 shows the agencies FLSI ranking per year and is color coded to 
show top four ranks and the bottom four ranks. 


^ For the purpose of this study, ANOVA was used to analyze differenees in means between ageneies. 
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Table 5. FLSI Agency Rank 


FLSI Rank 

Rank 

2006 

2008 

2010 

1 

ATF 

FBI 

EBI 

2 

DBA 

DEA 

DEA 

3 

FBI 

CG 

ATF 

4 

SS 

ATE 

CG 

5 

CG 

CIS 

SS 

6 

CIS 

SS 

USMS 

7 

USMS 

USMS 

CIS 

8 

ICE 

ICE 

FEMA 

9 

FEMA 

EEMA 

ICE 





10 

CBP 

TSA 

CBP 

11 

TSA 

CBP 

TSA 


As discussed earlier, perceptions observed in survey data “have shown that 
measures of perceived organizational performance correlate positively, with moderate to 
strong association, with objective measures of organizational performance” (Brewer, 
2005, p. 511). Although, for the years studied, the difference between the least effective 
and most effective agencies, as defined by the FLSI, is no more than four points, this still 
represents a significant statistical difference since in each year there were over 9,000 
nonsupervisor respondents to the survey for the agencies examined.^ Therefore, the 
aggregate mean is sufficient in ranking the agencies as most and least effective. 

1. Correlation of Behavior and Outcome Factor 

Using Pearson correlation,^ it was found that the behavior factor (the aggregate of 
behaviors described in the FSLI) and outcome factor (the aggregate of outcomes 
described in the FLSI) were significantly correlated above the R=0.701 level for each 
agency and for each year included in this study. Additionally, when combined into the 
FLSI, a strong positive correlation existed above R=0.701 level for each factor when 

^ To change an agency’s rank, several hundred respondents would need to respond differently. 

9 Pearson correlation is a statistical tool that tests linear dependence between two variables. 
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examined separately. This suggests that behaviors and outeomes of supervisors are 
elosely eonneeted. Therefore, it follows that if one assesses and seleets supervisors based 
on speeific supervisor behaviors, then supervisor outeomes will be impaeted. The 
eorrelation between behaviors and outeomes is very important and should strongly be 
eonsidered when ageneies are designing and implementing assessment methods for first- 
line supervisors. For details on the Pearson eorrelation results, see Appendix D. 

2. FLSI Trends 

Over the three years examined during this study, some interesting trends ean be 
observed for the 11 ageneies ineluded. First, FBI, DBA, CG and ATF have eonsistently 
been in the first four spots for the FLSI; however, in 2006, the SS ranked number four 
and the CG ranked number five. 

Seeond, there was slight movement with regard to rank order for the bottom four 
ageneies, but the same four ageneies remained in the bottom for all three years. Although 
there eould be several reasons for the ageneies falling at the bottom of this ranking, to 
inelude first-line supervisor seleetion proeesses, it should be noted that these four 
ageneies, ICE, FEMA, CBP and TSA, were all heavily impaeted or reorganized by the 
stand-up of the DHS in 2002. 

Eigure 1 shows the bottom four ageneies seem to be improving as it relates to the 
ELSI, with the largest movement of over three points by TSA. Eor TSA, this eould be a 
refieetion of the stabilization of first-line supervisor seleetion proeesses following the 
ineeption of the ageney in 2002. Although there is generally a slight movement upward 
for the top four ageneies, all four ageneies had less than one point inerease per year. 
Perhaps this is an indieation of eonsisteney and effeetiveness of first-line supervisor 
seleetion proeesses in these ageneies. Additionally, none of the 11 ageneies examined 
exhibited a downward trend as defined by the ELSE Although somewhat unremarkable, 
this eould demonstrate that none of the ageneies examined have made signifieant ehanges 
in their first-line supervisor seleetion proeesses that adversely impaet the effeetiveness of 

10 When these agencies were reorganized, first-line supervisor selection criteria may not have been 
clear and sometimes irrelevant as more senior employees from other agencies were integrated into these 
agencies—pre-empting a rational selection process. 
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it first-line supervisors. Lastly, Figure 1 demonstrates that in all three years ineluded in 
this study, the top four ageneies, as defined by the FLSI, seored above the overall FLSI 
government mean. 11 This perhaps provides eredence to the overall effectiveness of first- 
line supervisors for the top four agencies and provides reason to look to these agencies 
for best practices in first-line supervisor selection. 

il i- 



FBI DEA <6 ATF S$ USAtS CIS FEt.lA ICE CBF> TSA 


Figure 1. Mean Rating of Agencies on Supervisor Index 


11 Using all nonsupervisor survey data from each year examined, the researcher was able to determine 
the overall FLSI government mean. The overall mean in reflected in the Figure 1. 
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F. INTERVIEWS/POLICY REVIEWS 

Based upon the results of the FLSI (see Table 5), “savvy in s iders” from the three 
agencies in the most effective group and three agencies in the least effective group were 
interviewed, or, in the alternative, first-line supervisor selection policies were reviewed. 
The agencies examined in this way were: 

Most effective: 

• Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) 

• Drug Enforcement Administration (DEA) 

• Eederal Bureau of Investigation (EBI) 

Eeast effective: 

• Immigration and Customs Enforcement (ICE) 

• Customs and Border Protection (CBP) 

• Transportation Security Administration (TSA) 

Although the Coast Guard (CG) ranked in the most effective group for the EESI, 
it was not examined because it was considered an outlier for purposes of this analysis. It 
was treated as an outlier because other than the fact that CG falls within Department of 
Homeland Security (DHS), as an agency, it is more like Department of Defense (DoD) 
agencies when selecting first-line supervisors. Eike other DoD agencies, the CG has an 
extensive vetting and selecting process for first-line supervisors in both the officer and 
enlisted ranks. The CG processes are more holistic and akin to private company selection 
processes where development and selection is a valued practice and not a process defined 
solely by set times and set assessment tools. Although, like private companies, the 
military can provide significant insights to best practices for selecting first-line 
supervisors, its processes are so different from other agencies examined in this study, that 
fair comparisons are not possible. 
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G. CHARACTERISTICS OF SELECTION PROCESSES THAT TEND TO BE 

USED IN FEDERAL AGENCIES WITH THE HIGHEST AND LOWEST 

SUPERVISORY RATINGS 

Using interviews and poliey reviews, a seleetion proeess matrix was eompleted to 
determine similarities and differenees among the ageneies that were eonsidered in this 
study to be the most and least effeetive as defined by the FLSI. Table 6 shows the range 
of assessment tools used by the ageneies who partieipated in this study. Although there 
are other assessment tools that are reeommended by 0PM, none of the partieipating 
ageneies use these assessment tools and, as a result, they are not ineluded in the matrix. 
Some of the reeommended 0PM assessment tools do not appear to be used by any of the 
ageneies examined inelude: standardized tests that evaluate an applieant’s reasoning and 
logie and referenee eheeks. 

The effeetive and less effeetive ageneies examined have several similarities and 
differenees with eaeh other whieh ean be eategorized into four areas 1) use of self- 
assessment tools, 2) online administration of assessment tools, 3) interactive 
demonstration of behavior associated with desired competencies being sought and 4) use 
of multiple assessment tools. A comparison of the six agencies follows: 

Table 6. Results: First-Line Supervisor Selection Assessment Tool Matrix 


First-Line Supervisor Selection Assessment Tools 


Assessment Tool 

ATE 

DEA 

FBI 

ICE 

CBP 

TSA 

Non-Law Enfor 

Special Agent 

Non-Law Enfor 

Special Agent 

Non-Law Enfor 

Special Agent 

Special Agent 

Officer/Agricul 

Border Patrol 

Non-Law Enfor 

Air Marshal 

Pre-selection Vetting 




E 

1 

1 






Number of Hurdles in Selection Process* 

1 

1 

1 

2 

2 

2 

2 

1 

1 

2 

2 

Self-Assessment Portion (Y/N) 

N 

N 

N 

N 

N 

N 

Y 

Y 

Y 

Y 

Y 

Past Performance Evaluations 








Accomplishment Record 

E 


E 


E 

E 


E 

E 

E 

E 


Training & Experience 


AC 


AC 







E 

Practical Application Evaluations 








Situational Judgment Tests 


AC 


AC 


T 

E 

E 

E 
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Work Sample/Simulations 


AC 


AC 


T 

E 

E 

E 




Writing Skills 


AC 


AC 



E 

E 

E 



Job Knowledge Tests 


AC 


AC 

E 

T 

E 


E 



Interviews 

1 

AC 

1 

AC 

1 


1 



1 



*See Appendix B 





Key 

E=Electronic 









AC=Assessment Center 









l=ln-Person 









T= Telephonic 


1. Use of Self-Assessment Tools 

Tools requiring self-assessment were heavily used by those ageneies in the least 
effective group. CBP, ICE and ISA require applicants to self-assess their competencies 
based on several multiple choice questions within the assessment tools used. Based on 
the available responses on the self-assessment questions reviewed, an applicant can 
determine what answer will gamer a higher score. These scores are used as part of the 
overall first hurdle score, or in some cases, as the only score that is used to determine 
which applicants make it to the next hurdle. Therefore, it is possible that applicants will 
inflate their experience when responding to these self-assessment questions so they can 
continue to be considered for selection in the process. Moreover, applicants who assess 
themselves honestly and are potentially great first-line supervisors may not move to the 
next step of the selection process. The agencies examined that fell in the most effective 
group did not use any form of multiple choice self-assessment tools. Instead these 
agencies use assessment tools that allow for the observation and assessment of patterns of 
frequent behaviors by applicants. This appears to be a significant difference and might 
be a reason agencies have more or less effective first-line supervisors. 

2. Online Administration of Assessment Tools 

In every agency examined in this study, it appears that there is some aspect of the 
first-line supervisor selection process that is administered on a computer or over the 
Internet. For example, the initial submission of promotion packages appears to be 
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universally completed over the Internet. The Internet has streamlined the application 
process and accountability of applications; however, there are some interesting findings 
when agencies use computer-based assessment tools. 

Two of the three agencies that have the least effective first-line supervisors, ICE 
and CBP, use only computer-based assessment tools in their selection processes (see 
Table 6). Although these agencies use several assessment tools within the selection 
process, they do not use any assessment tools that are administered in a manner that can 
be observed by evaluators. Their processes appear to rely strictly on the responses an 
applicant provides on the computer. Conversely, other than the initial submission of the 
application, none of the most effective agencies rely only on computer-based assessment 
tools. Typically, the effective agencies use a combination of computer-based assessment 
tools and in-person interactive assessment tools. This multi-method approach is assumed 
by the author to provide a more complete assessment score for the applicants. 

3. Interactive Demonstration of Behaviors Associated With 
Competencies Being Sought 

The data suggests that in-person interactive assessment tools are important to 
effective first-line supervisor selection processes. DEA and ATE have highly rated first- 
line supervisor selection processes based on the EESI ratings each year, and they were the 
only agencies examined that used formal assessment centers. Each agency’s assessment 
center administered slightly different assessment tools, but they both had requirements to 
demonstrate valued supervisor competencies and behaviors in practical exercises and 
simulations. Although the three agencies that were examined in the least effective group 
had some practical exercises, like an in-box assessment,!^ these assessment tools were 
administered online and had no interaction with role players or evaluators. The DEA, 
ATE and EBI each had a portion of their selection process that required interaction with 
role players and evaluators during the administration of the selection process. The 
interaction with role players and evaluators appears to be a significant difference in the 


An in-box assessment is an assessment tool used to simulate the prioritization of tasks and 
assignments first-line supervisors must eonduet as part of their supervisory duties. 
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most and least effective agencies. This finding provides important insight into what 
elements of selection processes appear to produce effective first-line supervisors. 

4. Use of Multiple Assessment Tools 

One similarity among all agencies examined was the use of multiple assessment 
tools; however, there are differences in the way the multiple assessment tools are used. 
More effective agencies appear to use multiple assessment tools to confirm behaviors 
associated with required competencies. For example, ATF and DBA both use the 
training and experience evaluation that systematically reveals and credits an applicant’s 
past experience. ATF and DBA then use several other assessment tools like situational 
judgment tests and simulations to confirm or discredit the finding from the training and 
experience evaluation. Conversely, TSA, specifically the Air Marshal Service, appears to 
use two assessment tools that reveal the same information in a similar manner. The 
Bederal Air Marshal Service (BAMS) relies on accomplishment records and on training 
and experience evaluations. Both assessment tools rely on information provided by the 
applicant through essay style answers and resumes. So, unlike the mutually validating 
methods described above, by using this method, BAMS appears unable to confirm that 
the applicant’s experience meets the competencies being sought. in this light, one 
could argue for the elimination of one of these assessment tools to save overall 
assessment costs. Since the less effective agencies had other factors mentioned above, 
like computer-based assessments and self-assessment portions that appear to have 
impacted their first-line supervisor selection process effectiveness, it is not possible to 
determine the benefit of using multiple assessment tools in their selection process. 
However, since effective agencies use multiple assessment tools and tend not to rely 
solely on computer-based assessment tools and self-assessment portions in the selection 
process, one can surmise that the use of multiple assessments tools that confirm applicant 
competencies in different ways could enhance the effectiveness of a first-line supervisor 
selection process. 


In 2011, FAMS is expected to revise its first-line supervisor selection process, incorporating 
additional assessment tools that better identifies the best qualified first-line supervisors in this organization. 
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5. 


Other Observations 


a. Agencies Impacted by the Establishment of DHS 

Following the development of the Department of Homeland Seeurity in 
2002, several agencies underwent significant reorganization. CBP, ICE, TSA and FEMA 
were all significantly impacted by the development of DHS. These agencies all were in 
the less effective FESI group but demonstrated increased effectiveness over the three 
years examined (see Figure 1). This is most evident with TSA, which scored over three 
points higher from 2006-2010.1"^ Although none of the agencies interviewed in this 
study have significantly modified their first-line supervisor selection process since 2005, 
it is possible that the impact of the stand-up of DHS on CBP, ICE, TSA and FEMA has 
affected the effectiveness of first-line supervisor’s selection processes in each agency and 
the increased effectiveness on the FESI is an indication of the stabilization of these 
impacted agencies. However, it can also be argued that each of these agencies have had 
almost a decade to improve their processes and therefore, at a minimum, should see 
greater increase in first-line supervisor effectiveness over the period this study focuses 
on. CBP, ICE, TSA and FEMA all fall well under the government-wide FESI mean 
compared to other agencies examined like SS, USMS and CIS. Because of this, it 
appears unlikely that the changes caused by the standup of DHS were the only reason for 
these agencies having less effective first-line supervisor selection processes. 

b. Leveraging Common Assessment Tools 

The implementation of each assessment tool has a cost burden associated 
to it. As depicted in Table 1 in Chapter II, 0PM provides general insight into the 
financial burden each assessment may have on an agency. Interviews revealed that CBP 
and ICE appear to use innovative ways to lower costs for each agency. It appears that 
CBP and ICE leverage human resource personnel and common assessment tools to 
counter act these financial burdens. For example, for most first-line supervisor selection 
processes in CBP and ICE, one human resource office handles all aspects of the selection 

Although an increase of three points does not appear significant, it is important to note that the 
nonsupervisor survey respondents for TSA increased from 448 in 2006 to 1156 in 2010. This increase is 
significant and makes TSA’s three point increase important. 
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process. This includes information sheets for perspective applicants and the development 
and selection of assessment tools used in each process that address competencies being 
sought. This sharing of resources lowers the overall costs for each agency and creates 
consistency among agencies; however, the sharing of resources appears to negatively 
impact first-line supervisor selection processes when assessment methods employed are 
less effective since the processes and assessment methods impact more than one agency. 
In essence, perhaps it is the assessment tools used, not the sharing of human resource 
personnel, which has impacted the FLSI rank for CBP and ICE. When effective 
assessment methods are used by agencies that share human resource personnel, the first- 
line supervisor effectiveness will increase while minimizing the cost to each agency. 

c. Use of Probationary Period 

As indicated in the Chapter II, 0PM suggests that the use of the 
probationary period is important and vital aspect of the selection process for government 
employees, specifically first-line supervisors. The interviews and policy reviews 
revealed that none of the agencies examined appeared to consider the probationary period 
to be part of the selection process. This is consistent with past studies that indicated that 
the probationary period was not used to weed out first-line supervisors that were not 
meeting performance standards and goals for first-line supervisors in each agency. As 
agencies struggle to select the most effective first-line supervisors, it appears that they 
should look to leverage the probationary period as a final assessment tool in their first- 
line supervisor selection process. 

H. CONCLUSIONS 

The survey data, interviews, and policy reviews conducted as part of this study 
provide significant insight into what makes an effective first-line supervisor selection 
processes in the federal government. It appears that the agencies examined follow the 
laws and regulations when selecting first-line supervisors; however, because 0PM allows 
for substantial flexibilities in assessment methods used by each agency, the effectiveness 
of first-line supervisor selection processes is inconsistent. Additionally, the use of self- 
assessment tools, computer-based assessment methods and assessment tools that focus on 
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past performance, not current behavior, do not tend to produce effective first-line 
supervisors. Whereas, selection processes that use multi-method approaches like the use 
of in-person interactive assessment tools that assess desired behavior, appear to produce 
more effective first-line supervisors. Based on the results of data analyzed in this study, 
several recommendations can be made that perhaps could improve the first-line 
supervisor effectiveness in the federal government. This will be the subject of the next 
section. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


Not everything that ean be eounted eounts, and not everything that eounts 

ean be eounted. 


Albert Einstein 


A. OVERVIEW 

Bureaueraeies are needed beeause they provide stability, order and proteet 
liabilities (Beetham, 1974). As sueh, a bureaueraey like the federal government should 
not ehange polieies and proeedures quiekly or haphazardly. By doing so, the stability the 
bureaueraey affords would be jeopardized. But in order for the federal government to 
keep up with ehanges in soeiety and new threats, it has to be willing to inerementally 
adopt and update polieies and proeedures. This ean be done through eommon sense 
reeommendations that impaet speeifie polieies and proeedures like those affeeting first- 
line supervisor seleetion. 

The role and importanee of first-line supervisors in the federal government has 
drastieally inereased sinee 9/11; however, many of the polieies, proeedures and proeesses 
that govern the seleetion of this eritieal asset are outdated and ineffeetive (Human 
Resourees Management Panel, 2003; O’Driseoll & Whitehouse, 2005; Offiee of Merit 
Systems Oversight and Effeetiveness, 2001). These polieies, proeedures and proeesses 
push federal ageneies to avoid doing wrong instead of pushing ageneies to do right in the 
seleetion of first-line supervisors. Although ageneies and hiring offieials need latitude 
when seleeting first-line supervisors, they should also be held aeeountable for the 
methods and proeesses that are used. Based on the results of the researeh eondueted in 
this study, several observations and reeommendations are provided. These observations 
and reeommendations are intended to provide insights and potential best praetiees to 
federal ageneies, human resouree speeialists and seleeting offieials who are interested in 
improving the quality of first-line supervisors in the federal government. 
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B. PREDICTORS OF EFFECTIVE FIRST-LINE SUPERVISORS 


It has long been argued that the best predietor of future performanee is past 
performanee. This is why most hiring or seleetion proeesses require evidenee of past 
experienee and training, often in the form of a resume or eompeteney narratives. Today, 
many applieants hire professional resume and narrative writers, and organizations with 
limited resourees use self-assessments as first hurdles in the selection process instead of 
verifying an applicant’s background and experience. One might question if this is this 
the best way to select first-line supervisors. Additionally, these assessment tools usually 
reveal an applicant’s technical competencies rather than competencies and leadership 
potential that are needed to be an effective first-line supervisor. Unlike many private 
companies that observe and evaluate employees for leadership potential as part of a 
development program that starts when they are first hired (Charan, 2008; Charan, Drotter, 
& Noel, 2001), the federal government tends to evaluate employees not on what they 
have the capability to do, but rather what they have done in the past. This can be a faulty 
basis on which to evaluate employees because past experience may have little to do with 
leadership potential or success as a supervisor. While effective first-line supervisors 
might be selected by this method through happenstance, the federal government is better 
served by purposefully selecting the best candidates for first-line supervisor jobs. 

It is unlikely that federal agencies will be able to accurately and fairly create an 
environment similar to private companies where leadership development is an 
evolutionary and deliberate process. Constant turnover in agencies and governing laws 
and regulations make this difficult, if not impossible, in the federal government. So the 
federal government must build into its first-line supervisor selection process steps that 
will best position agencies for selecting the most effective first-line supervisors. These 
steps cannot be solely the agency’s responsibility. They have to be woven into several 
layers of the government from the selecting official to Office of Personnel Management 
(0PM). This will ensure the selection processes for first-line supervisors are fair and 
consistent, reliably produce effective first-line supervisors, and are resistant to changes in 
Presidential Administrations and senior political appointees leading agencies. 
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To anchor the selection processes in all levels in federal organizations, the 
government must fundamentally change the way they view and assess eompeteneies 
required by first-line supervisors. Clearly, past experienee and training is an important 
aspeet of any applieation; however, the government eannot allow ageneies to solely use 
an applieant’s reported baekground and experienee. It is merely a first hurdle in a 
seleetion proeess. As Buekingham and Coffman (1999) suggest, past performanee alone 
is not an indieator of future sueeess. Predietive power eomes from 2 l frequent pattern of 
the behavior. This predieative power only ean be uneovered through assessment tools that 
reveal the frequent and demonstrated patterns of behavior that are being sought as part of 
the designated eompeteneies. This study suggests specifie aetion that should be taken by 
any agency or leader who wants to produce effeetive leadership at the first-line 
supervisor level. 

C. THE ROLE OF OPM IN FIRST-LINE SUPERVISOR SELECTION 

As the human resouree ageney within the federal government, OPM should 
assume a more aetive role in guiding design of the seleetion proeess of first-line 
supervisors. That is, they should require ageneies to do more than just rely on past 
experienee and training. On May 11, 2010, President Obama issued a federal hiring 
reform memorandum. The guidanee in the memorandum called for modifying the hiring 
and reeruiting of federal employees. Key modifieations were: 

• Elimination of written essays (KSAs) 

• Allow individuals to apply with resume and eover letter 

• Use eategory rating 

• Ensure manager responsibility and aeeountability for hiring 

• Improve quality and speed of hiring (Obama, 2010) 

Although this memorandum is not direeted solely at first-line supervisor seleetion, 
it demonstrates the Administration’s coneem and foeus on federal human eapital 
aetivities. The established reform guidanee and subsequent aetions taken by OPM to 
ensure this guidanee is implemented represents signifieant steps in improving the federal 
workforce; however, if one reviews the Eiscal Year 2010 Agency Einancial Report 
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published by 0PM, it is clear that although 0PM is focused on the federal hiring reform, 
the primary focus of 0PM does not include first-line supervisor selection (United States 
Office of Personnel Management, 2010). By not putting more emphasis on first-line 
supervisors, the reforms that President Obama is requiring will fail. Effective first-line 
supervisors are needed to mentor, lead, and provide guidance to the federal employees 
President Obama is targeting with this hiring reform. Concurrent to the reforms required 
by the President, 0PM must examine and move to reform its role in the selection 
processes at the first-line supervisor level, if it wants to meet the second part of the 
agency’s strategic mission of retaining and honoring the federal workforce. This is 
because studies have shown that employees will stay in an organization primarily based 
on their supervisor. Hence, there is more at stake than selection and development of first- 
line supervisors. Retention of the workforce is at risk as well. Specifically, in a 2004 
Corporate Leadership Council study, 50,000 employees worldwide indicated that 22 of 
the top 25 drivers of employee’s retention at their company were supervisor-led 
(Thompson, 2007, p. 3). 

It is unrealistic for 0PM to completely take over or overhaul the first-line 
supervisor selection process in a centralized manner; however, by refining first-line 
supervisor selection guidance, 0PM can effect change in first-line supervisor selection 
processes without taking control and autonomy away from the agencies that 0PM serves. 

While 0PM, as discussed in Chapter II, provides extensive assessment tools and 
resources to federal agencies, interviews and policy reviews conducted during this study 
suggest 0PM leaves the decision about use of these tools to the various agencies. 
Although this decentralized approach provides enormous flexibility to agencies to 
develop first-line supervisor selection processes customized to an agency’s unique needs, 
it creates an environment that does not cultivate effective selection processes needed or 
selection consistency. 

Perhaps 0PM should implement a system that provides more stringent regulations 
for the assessment and selection methods used to select first-line supervisors. Using best 
selection practices, some of which were examined in this study, 0PM could provide 

minimum assessment method options based on constraints identified by each agency. 
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Instead of just informing agencies of the assessment tools pros and cons based on agency 
restraints, 0PM should require minimum assessment methods that are needed based on 
certain restraints. For example, if an agency has several hundred possible applicants for a 
supervisory position that needs to be fdled within 60 days, perhaps 0PM could provide a 
minimum set of assessment tools that agencies should chose from that would accomplish 
their goal; yet at the same time, ensure selection processes that have the best opportunity 
of identifying the best qualified applicants and that would prevent agencies from only 
using self-assessments and competency narratives in evaluating applicants. 

By creating minimum selection process criteria, 0PM would take a substantial 
step in ensuring first-line supervisor selection processes across agencies are more 
effective than the processes that are currently used. This minimum selection criteria 
would allow for agencies to continue crafting selection processes to fit their individual 
needs but create a level of consistency in the federal government that will strengthen its 
foundation—the first-line supervisors. 

D. CREATING CONNECTEDNESS WITHIN DHS 

This study revealed agencies that have the least effective first-line supervisor 
selection processes are those impacted by the stand up of DHS, specifically FEMA, CBP, 
ICE and TSA. This is one more piece of evidence that indicates many of the agencies that 
fall under DHS struggle to stabilize and create effective processes. Clearly, as many 
Government Accountability Office (GAO) reports have indicated, there are several 
potential steps that will create more stability and consistency for the agencies that fall 
under DHS (U.S. Governement Accountability Office, 2008; 2010). 

Most of the recommendations provided by GAO are at a strategic level such as 
implementing management functions and strengthening performance measurements in 
DHS. Since DHS agencies tended to score lower on perceptive measures of effectiveness 
(the EESI), perhaps DHS can make strides in accomplishing these strategic 
recommendations by focusing on a specific process such as the first-line supervisor 
selection process, one that impact all agencies within DHS. As DHS continues to 
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establish itself as a eabinet level ageney and as the eulture builds within this ageney, it is 
eritieal to establish eommon threads and standards, wherever possible, fostering a unified 
DHS. 

A eommon first-line supervisor assessment proeess would be an important step in 
ereating a more eonneeted DHS eulture in whieh ah the sub-ageneies use the same 
proeess to seleet employees for leadership responsibility. Similar to the Treasury 
Department’s Treasury Enforeement Agent Exam (TEA) and Department of Defense’s 
Armed Serviees Voeational Aptitude Battery (ASVAB), DHS should ereate a eommon 
assessment test that ah potential first-line supervisors within DHS must pass. Granted, 
the TEA and ASVAB are administered for entry-level positions, but what is important 
and ean be applied to the first-line supervisor seleetion proeess is the eonsisteney and a 
eommon thread that the ASVAB and TEA ereates. Regardless of the military braneh or 
military speeialty, every applieant must take the ASVAB. Eaeh military braneh 
determines how to use the ASVAB seore to determine the military speeialty for whieh the 
applieant is eligible. This eommon test is required by ah military branehes and ereates 
eonsisteney and a eommon evaluation standard. By employing a eommon first hurdle 
assessment test for first-line supervisors, DHS will ereate more eonneetedness within the 
department. 

As diseussed in the results seetion of this study, the use of assessment tools varies 
greatly among the ageneies examined. Although the ageneies need latitude to strueture 
the seleetion proeesses that best meets the eompeteneies and qualifieations needed, there 
are benefits to having a baseline hurdle that ah first-line supervisors must go through in 
order to be eonsidered for speeifie positions in a given ageney. Eor example, there are 
several Psyehometrie tests like eognhive abilhyi^ and emotional intelhgeneei^ tests that 
ean be administered online and eould provide several benefits to DHS, its sub-ageneies 


Cognitive ability tests test abilities involved in thinking (reasoning, pereeption, memory, verbal, and 
mathematieal ability and problem solving). 

Emotional intelligenee (El) test is defined as a type of soeial eompetenee involving the ability to 
monitor one’s own and other’s emotions, to diseriminate among them, and to use the information to guide 
one’s thinking and aetions. El is fairly speeifie ability that eonneets a person’s knowledge proeesses to his 
or her emotional proeesses. 
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and selecting officials. These psychometric tests evaluate an applicant’s abilities 
involved in thinking, to include reasoning, perception, memory, and problem solving and 
how these abilities are connected to his or her emotional processes. A standard 
psychometric test would provide the following benefits to the agencies that fall within 
DHS: 

1. Provide valuable demographic information to 0PM, DHS and sub¬ 
agencies that could be important in analyzing the federal workforce. 

2. Increase face validityi^ of first-line supervisor’s selection processes across 
the federal government, which impacts overall morale and support of first- 
line supervisors. 

3. Create a first-line supervisor competency baseline that gives insight into 
competencies being sought by each agency. 

4. Provide a strand of unity among all first-line supervisors. Therefore, if a 
first-line supervisor desires to changes agencies, the receiving agency 
could use these scores as a first step in determining suitability. 

5. Remove part of the financial burden from agencies and ensures the 
reevaluation and modification of such assessment tests could be uniformly 
updated and applied across the federal workforce in DHS. 

This first-hurdle assessment test would not require a mandatory elimination of 
potential first-line supervisors based on a set minimum score. Instead, similar to the 
military and the Treasury Department, the agency the applicant is applying for would 
receive the complete list of names with the scores from the psychometric test(s). It would 
be up to each agency to determine how these results would factor into the first-line 
supervisor process they have in place. For example, if an agency uses self-assessments, 
interviews or situational judgment tests in the selection process, the psychometric test 
results could be compared to these results to validate findings from these assessment 
tools. This will provide insights into specific applicants that an agency might want to 
further investigate. 

With over 200,000 employees, the DHS workforce is large, which makes it hard 
to create and ensure consistency in any human resource function. By creating an online 

Face validity is how a process appears at “face value.” Does a process appear to select first-line 
supervisors in a fair and unbiased way? 
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centralized common first-hurdle test for first-line supervisors, perhaps this would assist 
DHS in creating eonsisteney in first-line supervisor selection processes and begin to 
establish overall connectedness within the department. 

E. SELECTION PROCESS BEST PRACTICES 

As stated before, this study is not intended to suggest a eookie-eutter selection 
process for seleeting effective first-line supervisors. This would discount several factors 
that influence the effectiveness of first-line supervisors that are outside the scope of the 
selection process. But the results of this study suggest there are several best practiees that 
surface for seleeting first-line supervisors. 

1. Avoid Self Assessment 

Self-assessments produce potentially flawed results. If an agency uses self- 
assessments, they should be used as an assessment tool only in the first hurdle of the 
seleetion proeess. Most applicants know that to be considered and make it through the 
first hurdle of a selection process, a certain score has to be obtained through self- 
assessment. Therefore, there is no drawbaek for an applieant to inflate the rating. Self- 
assessment tools allow for dishonest applicants to make it to the next hurdle with little 
fear of being held accountable; whereas honest, potentially great first-line supervisors 
may not make the cut. This dramatically impacts morale in an organization since the 
self-assessments may be prone to having unqualified applieants make it to the next 
hurdle, whereas potentially qualified applicants may get disqualified and are unable to 
advanee. This study reveals that the ageneies that were rated as less effective relied on 
self-assessment as part of their first hurdle for selection of first-line supervisors. Doing 
away with self-assessments tools, agencies eould save money or invest in more valid 
assessment tools. 

2. Applicant Interaction Assessment Tools 

The ageneies defined as effective in this study all had some applicant interaction 
assessment tools used in their seleetion proeess. The FBI’s interaction was through a 
telephone administered in-box assessment tool, whereas the DBA and ATF both used 
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assessment eenters that eontained multiple assessment tools, many requiring in-person 
observations of desired eompeteneies and behaviors. Observation assessment tools sueh 
as interviews, praetical exereises and simulations, provide the eonnection between what 
an applicant has the potential to do based upon his or her training and experience, and 
what an applicant can currently demonstrate. This allows evaluators and assessors to 
observe current behaviors that actually apply to required and important supervisor 
competencies. Additionally, more applicant interactive assessment tools could help 
eliminate the federal government’s habit of selecting first-line supervisors based on 
technical expertise instead of supervisor capabilities and competencies. 

3. Multi-Method Approach 

More than one selection assessment tool is needed in any first-line supervisor 
selection process in the federal government. Although the results of this study seem 
somewhat inconclusive regarding the number of hurdles and assessment tools needed, it 
is clear agencies with more effective first-line supervisors (rated by the FLSI) used 
multiple assessment tools, did not rely solely on self-assessments and used assessment 
tools that required applicant interaction. Additionally, this multi-method approach is 
consistent with recommendations from 0PM and the Department of Labor (United States 
Department of Labor, 2000; U.S. Office of Personnel Management, 2007). In short, this 
approach provides the clearest picture of a candidate’s abilities in terms of competencies 
being sought because is reveals frequent patterns of behavior. Most private companies do 
not have set selection hurdles or tools in their first-line supervisor selection processes 
since they evaluate potential first-line supervisors over a period of time instead of a set 
selection process. However, since the federal government uses a set selection process, in 
order to get the best understanding of the applicant’s competencies, agencies should use a 
multi-method approach in evaluating applicants. Additionally, when deciding on the 
assessment tools to use, agencies should select assessment tools that complement each 
other and not just provide the same information in a different or disjointed manner. This 
multiple hurdle and multiple assessment tool approach allows for the evaluation of 
applicants by multiple evaluators and methods. Lastly, this approach provides more 

process validity, which is important for morale of current and future applicants, as well as 
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the employees who will report to these new supervisors. If an ageney hopes to recruit the 
best employees for first-line supervisor positions, the process must be fair, valid, 
transparent and select the best qualified candidates. 

F. MEASURING FIRST LINE SUPERVISORS 

H. James Harrington stated; 

Measurement is the first step that leads to control and eventually to 
improvement. If you can’t measure something, you can’t understand it. If 
you can’t understand it, you can’t control it. If you can’t control it, you 
can’t improve it. 

Perhaps the government’s inability to measure first-line supervisor performance is 
the reason why the effectiveness of first-line supervisors in the federal government 
continues to be called into question. Although the best practices mentioned above will 
potentially assist in selecting effective first-line supervisors, in order to allow agencies to 
determine if changes to first-line supervisor selection processes are working, there must 
be a method to gauge and monitor the effectiveness of first-line supervisors. 

This study used the perceptions of nonsupervisory employees, measured by the 
First-Line Supervisor Index (FLSI), to determine the effectiveness of first-line 
supervisors in the agencies examined. Although the validity of perceptive measures can 
be questioned, these tools have been shown to have a high positive correlation to other 
measures of performance that can be used in examining the effectiveness of first-line 
supervisors. As described in Chapter II, a multi-method approach is perhaps the most 
valid method to measure performance and effectiveness; however, when examining first- 
line supervisors in the federal government, there is currently no other data available that 
can be used that examines effectiveness of first-line supervisors at the micro level. 
Therefore, potential questions about the usefulness of perceptive measures should not 
stop the federal government from using these perceptions to measure performance and 
effectiveness of first-line supervisors. 

The FLSI used in this study was a combination of eight questions (see Table 7 

and Appendix A) that were consistently used in the Federal Human Capital Survey 

(FHCS) in 2006 and 2008, in the Federal Employee Viewpoint Survey (FedView) in 
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2010 and focused on specific behaviors and outcomes of first-line supervisors. In 2010, 
as part of the FedView, 0PM created a “My Supervisor/Team Leader” sub-index that 
eontained the eight questions consistent throughout the three examined in this study and 
three additional questions. 


Table 7. Supervisor/Team Leader Questions 


My Supervisor/Team Leader Sub-Index 

2006/2008 

2010 

My supervisor supports my need to balance work and other life issues. 

V 

V 

My supervisor/team leader provides me with opportunities to demonstrate my 
leadership skills 

V 

V 

Discussions with my supervisor/team leader about my performance are 
worthwhile. 

V 

V 

My supervisor/team leader is committed to work force representative of all 
segments of society. 

V 

V 

My supervisor/team leader provides me with constructive suggestions to 
improve my job performance. 

V 

V 

Supervisors/team leaders in my work unit support employee development. 

V 

V 

My supervisor/team leader listens to what I have to say. 


V 

My supervisor/team leader treats me with respect. 


V 

In the last six months, my supervisor/team leader has talked with me about 
my performance. 


V 

I have trust and confidence in my supervisor. 

V 

v 

Overall, how good a job do you feel is being done by your immediate 
supervisor/team leader? 

V 

a/ 


By using the 11 questions from “My Supervisor/Team Leader” sub-index, 
agencies can begin to track and measure first-line supervisor performance and 
effectiveness. Since 0PM has begun administering the FedView every year starting in 
2010, agencies can track first-line supervisor performance and effeetiveness as part of a 
yearly review. This will provide immediate feedback and insight to any changes that 
agencies might have made in the first-line supervisor selection processes or other policy 
changes that impact first-line supervisor performance and effectiveness within their 
agency. 


G. THE COST OF SELECTING EFFECTIVE FIRST-LINE SUPERVISORS 

During the interviews conducted for this study, human resource specialists 
expressed concern about the cost associated with selection processes. Many persons 
interviewed reported cost often impacted the choice of assessment methods used in first- 
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line supervisor seleetion. Reeent budget euts will eontinue to make the eost assoeiated 
with seleetion proeesses an issue of eoneern for ageneies. Therefore, DHS, whieh had 
the lower FLSI seores, must look for innovative ways to allow ageneies to use needed 
assessment tools but keep eosts down. Although there are numerous innovative ways to 
out eosts for seleetion proeesses, this study revealed two that are of note. 

First, by implementing a standard first hurdle psyohometrio assessment test, the 
eosts potentially oan be reduoed for ageneies within DHS depending on how they plan to 
use the test in the overall seleetion prooess. Seoond, DHS human resouroe 
representatives should ooordinate with eaoh other to share assessment methods they have 
in oommon. Although this study revealed that CBP and ICE have less effeotive first-line 
supervisors oompared to the other ageneies examined, a best praotioe in potentially 
saving money was revealed. CBP and ICE reduoe the eost of first-line supervisor 
seleetion by sharing oommon seleetion methods for first-line supervisors. This praotioe 
oould be emulated by all DHS ageneies as a strategy to out down eosts assoeiated with 
implementing best praotioes for first-line supervisor seleetion. Additionally, by sharing 
eosts, resouroes and assessment tools, the use of oommon best praotioe standards and 
oonneotedness among ageneies within DHS will be enhanoed. 

Clearly, the ageneies within DHS need to find ways to implement best praotioes to 
improve seleetion proeesses for first-line supervisors while keeping the finanoial burden 
for any one agenoy at a minimum. Senior exeoutives of eaoh DHS agenoy need to 
remember the potential eost of poor or ineffeotive first-line supervisor seleetion 
proeesses goes beyond a human resouroe budget allooation; that is; the loss of employees 
supervised by poor first-line supervisors. Ineffeotive seleetion of first-line supervisors, 
thus, oan have a oasoading effeot that is hard to quantify, but potentially very oostly 
beoause first-line supervisors are oruoial to the day-to-day operation of any organization. 
They are direotly responsible for day-to-day leadership, ooaohing, mentoring, job 
performanoe and retention of nonsupervisory employees (Human Resouroes Management 
Panel, 2003), and they ensure that the vision, guidanoe and direotion of exeoutive leaders 
beoome reality in an organization. It is reasonable to say that the eost paid by an agenoy 
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that selects ineffective supervisors, grossly outweighs the costs associated with 
implementing an effective first-line supervisor selection process. 


H. FUTURE RESEARCH 

Several important areas of further research should be pursued. First, researchers 
should try to improve the perceptive measures of first-line supervisor effectiveness 
(FLSI) used in this study or create a new, more effective measure that would allow first- 
line supervisor performance to be assessed on a regular basis. 

Correspondence between perceptive measures of effectiveness and more objective 
measures of effectiveness needs to be explored as well. Mapping out the links between 
specific behaviors and outcomes like effectiveness would be very helpful for designing 
first-line supervisor training activities. 

Researchers should also examine what effect substantial changes in first-line 
supervisor selection process might have on how an agency’s supervisors are rated on the 
FLSI. 

Lastly, first-line supervisor selection best practices should be identified for any of 
these follow-on research efforts. Application of these best practices in specific agencies 
should be favorably considered to improve the overall quality of supervision in the 
federal government. 
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APPENDIX A. FIRST-LINE SUPERVISOR INDEX SURVEY 

INFORMATION 


Table 8. First-Line Supervisor Index Questions 


First-Line Supervisor Index Questions 

Survey Question 
Number 


2006 

2008 

2010 


12 

12 

42 

My supervisor supports my need to balance work and other life 
issues. 

13 

13 

43 

My supervisor/team leader provides me with opportunities to 
demonstrate my leadership skills. 

31 

31 

44 

Discussions with my supervisor/team leader about my performance 
are worthwhile. 

33 

34 

45 

My supervisor/team leader is committed to a workforce 
representative of all segments of society. 

47 

48 

46 

My supervisor/team leader provides me with constructive 
suggestions to improve my job performance. 

48 

49 

47 

Supervisors/team leaders in my work unit support employee 
development. 

7 

7 

51 

I have trust and confidence in my supervisor. 

9 

9 

52 

Overall, how good a job do you feel is being done by your 
immediate supervisor/team leader? 


Table 9. First-Line Supervisor Demographic Characteristics 


Demographic Characteristics 

What is your supervisory status? (DSUPER) 

2010 

Nonsupervisor/Team Leader 

Supervisor 

Manager/Executive 

2006 

& 

2008 

Nonsupervisor 

Team Eeader 

Supervisor 

Manager 

Executive Note: All 2006 and 2008 data was re-coded to match the 2010 

DSUPER selection options 


57 



























Table 10. First-Line Supervisor Index Value Labels 


Value Labels For FLSI Items 

Question Num 

)er Per Year 

2006 

2008 

2010 


12 

12 

42 

5=Strongly Agree 

4=Agree 

3=Neither Agree nor Disagree 

2=Disagree 

HStrongly Disagree 

X=Do Not Know 

13 

13 

43 

31 

31 

44 

33 

34 

45 

47 

48 

46 

48 

49 

47 

7 

7 

51 

5=Strongly Agree 

4=Agree 

3=Neither Agree nor Disagree 

2=Disagree 
l=Strongly Disagree 


9 

9 

52 

5=Very Good 

4=Good 

3=Fair 

2=Poor 

HVery Poor 



Table 11. Ageney Survey Code for Federal Human Capital Survey and Federal 

Viewpoint Survey 


Survey Code 

Agency 

DBA 

Drug Enforcement Administration (DEA) 

DJ02 

Federal Bureau of Investigation (FBI) 

DJ08 

U.S. Marshals Service (EiSMS) 

DJ15 

Alcohol, Tobacco, Firearms, and Explosives (ATF) 

HSAB 

Citizenship and Immigration Services (CIS) 

HSAC 

U.S. Coast Guard (CG) 

HSAD 

U.S. Secret Service (SS) 

HSBB 

Immigration and Customs Enforcement (ICE) 

HSBC 

Transportation Security Administration (TSA) 

HSBD 

Customs and Border Protection (CBP) 

HSCB 

Federal Emergency Management Agency (FEMA) 
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APPENDIX B. FIRST-LINE SUPERVISOR SELECTION PROCESS BENCHMARKS 


Table 12. Benchmarks for First-Line Supervisor Selection Processes 


First-Line Supervisor Selection Process 


Hurdles 

ATF 

DEA 

FBI 

ICE 

CBP 

TSA 

N on-Law Enfor 

Special Agent 

N 0 n-Law Enfor 

Special Agent 

Special Agent 

N 0 n-Law Enfor 

Special Agent 

Officer/Agricul 

B ord er Patrol 

N on-Law Enfor 

Air M arshal 

1st Hurdle 

KSAand Resume 
Application Package 

Assessment Center- 

(Oral Presentation, 
Simulations, Panel 
Interview) 

KSAand Resume 
Application Package 

Supervisor Rating 

Leadership Skills 
Assessment In-Baske 

Simulation 

KSAand Resume 
Application Package 

Logic Reasoning Test 

Logic Reasoning Test 

Logic Reasoning Test 

KSAand Resume 
Application Package 

KSAand Resume 
Application Package 





Assessment Center- 
(Panel interview, 
Simulations, Written 
Test) 



Multiple Choice 
Writing Test 

Multiple Choice 
Writing Test 

Multiple Choice 
Writing Test 


Supvervisor Rating 


Computer-Based In- 
Basket Simulation 

In-Basket Simulation 

In-Basket Simulation 


Job-knowledge Test 

Career Experience 
Invetory 

Career Experience 
Invetory 



Job-Knowledge Test 

Administration 

Electronic 

Submission 

In-person/Computer 

Based 

Electronic 

Submission 

In-person/Computer 

Based 

3 Person Panel- 

Telephone 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 







2nd Hurdle 

Struture Interview 


Struture Interview 

HQ Career Board 

Competencies 

Narrative Reviewed 
bySES-Level Board 

Struture Interview 

Struture Interview 

HQ Career Board 

Administration 

In-Person at Field 

Office 

In-Person at Field 

Office 

HQ Panel 

HQ Career Board 

In-Person at Field 

Office 

In-Person at Field 

Office 

HQ 3 Person Panel 
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APPENDIX C. ANOVA TABLES/GRAPHS 


Table 13. ANOVA 2006 


ANOVA 2006 

Sum of 
Squares 

df 

Mean Square 

F 

Sig. 

FLSI Between Groups 

18534.990 

10 

1853.499 

32.925 

.000 

Within Groups 

388993.442 

6910 

56.294 



Total 

407528.432 

6920 













Table 14. ANOVA2008 


ANOVA 2008 

Sum of 
Squares 

df 

Mean Square F Sig. 

FLSI 

Between Groups 

13915.978 

10 

1391.598 23.209 .000 


Within Groups 

474997.396 

7922 

59.959 


Total 

488913.374 

7932 










Table 15. ANOVA2010 


ANOVA2010 

Sum of 
Squares 

df 

Mean Square 

F 

Sig. 

FLSI Between Groups 

10393.630 

10 

1039.363 

14.604 

.000 

Within Groups 

639606.256 

8987 

71.170 



Total 

649999.886 

8997 
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APPENDIX D 


FLSI CORRELATION TABLES 


Table 16. 2006 Correlation Tables 


DEA 2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.787(**) 

.982(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1175 

1175 

1175 

Outcome Factor 

Pearson Correlation 

.787(**) 

1 

.890(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1175 

1175 

1175 

FLSI 

Pearson Correlation 

.982(**) 

.890(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1175 

1175 

1175 



Behavior 

Outcome 


FBI 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.725(**) 

.973(**) 


Sig. (2-tailed) 


.000 

.000 


N 

422 

420 

420 

Outcome Factor 

Pearson Correlation 

.725(**) 

1 

.864(**) 


Sig. (2-tailed) 

.000 


.000 


N 

420 

432 

420 

FLSI 

Pearson Correlation 

.973(**) 

.864(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

420 

420 

420 



Behavior 

Outcome 


USMS 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.762(**) 

.980(**) 


Sig. (2-tailed) 


.000 

.000 


N 

530 

530 

530 

Outcome Factor 

Pearson Correlation 

.762(**) 

1 

.876(**) 


Sig. (2-tailed) 

.000 


.000 


N 

530 

530 

530 

FLSI 

Pearson Correlation 

.980(**) 

.876(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

530 

530 

530 


** Correlation is significant at the 0.01 level (2-tailed). 


The FLSI is the aggregate mean of eight questions concerning first-line supervisor behavior s and 
outcomes. The behavior factor is the aggregate mean of behaviors (six questions) and the outcome factor is 
the aggregate mean of the outcomes (two questions) in the FLSI. 
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ATF 2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.803(**) 

.983(**) 


Sig. (2-tailed) 


.000 

.000 


N 

557 

557 

557 

Outcome Factor 

Pearson Correlation 

.803(**) 

1 

.898(**) 


Sig. (2-tailed) 

.000 


.000 


N 

557 

557 

557 

FLSI 

Pearson Correlation 

.983(**) 

.898(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

557 

557 

557 



Behavior 

Outcome 


CIS 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.752(**) 

.98I(**) 


Sig. (2-tailed) 


.000 

.000 


N 

647 

647 

647 

Outcome Factor 

Pearson Correlation 

.752(**) 

1 

.866(**) 


Sig. (2-tailed) 

.000 


.000 


N 

647 

647 

647 

FLSI 

Pearson Correlation 

.981(**) 

.866(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

647 

647 

647 



Behavior 

Outcome 


USCG 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.804(**) 

.985(**) 


Sig. (2-tailed) 


.000 

.000 


N 

695 

695 

695 

Outcome Factor 

Pearson Correlation 

.804(**) 

1 

.895(**) 


Sig. (2-tailed) 

.000 


.000 


N 

695 

695 

695 

FLSI 

Pearson Correlation 

.985(**) 

.895(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

695 

695 

695 


** Correlation is significant at the 0.01 level (2-tailed). 


66 

















USSS 2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.764(**) 

.980(**) 


Sig. (2-tailed) 


.000 

.000 


N 

537 

537 

537 

Outcome Factor 

Pearson Correlation 

.764(**) 

1 

.878(**) 


Sig. (2-tailed) 

.000 


.000 


N 

537 

537 

537 

FLSI 

Pearson Correlation 

.980(**) 

.878(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

537 

537 

537 



Behavior 

Outcome 


ICE 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.758(**) 

.980(**) 


Sig. (2-tailed) 


.000 

.000 


N 

604 

604 

604 

Outcome Factor 

Pearson Correlation 

.758(**) 

1 

.872(**) 


Sig. (2-tailed) 

.000 


.000 


N 

604 

604 

604 

FLSI 

Pearson Correlation 

.980(**) 

.872(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

604 

604 

604 



Behavior 

Outcome 


TSA 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.804(**) 

.984(**) 


Sig. (2-tailed) 


.000 

.000 


N 

453 

448 

448 

Outcome Factor 

Pearson Correlation 

.804(**) 

1 

.896(**) 


Sig. (2-tailed) 

.000 


.000 


N 

448 

458 

448 

FLSI 

Pearson Correlation 

.984(**) 

.896(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

448 

448 

448 


** Correlation is significant at the 0.01 level (2-tailed). 
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CBP 2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.752(**) 

.980(**) 


Sig. (2-tailed) 


.000 

.000 


N 

847 

847 

847 

Outcome Factor 

Pearson Correlation 

.752(**) 

1 

.867(**) 


Sig. (2-tailed) 

.000 


.000 


N 

847 

847 

847 

FLSI 

Pearson Correlation 

.980(**) 

.867(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

847 

847 

847 



Behavior 

Outcome 


FEMA 2006 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.811(**) 

.983(**) 


Sig. (2-tailed) 


.000 

.000 


N 

461 

461 

461 

Outcome Factor 

Pearson Correlation 

.811(**) 

1 

.904(**) 


Sig. (2-tailed) 

.000 


.000 


N 

461 

461 

461 

FLSI 

Pearson Correlation 

.983(**) 

.904(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

461 

461 

461 


** Correlation is significant at the 0.01 level (2-tailed). 


Table 17. 2008 Correlation Tables 


DEA 2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.815(**) 

.984(**) 


Sig. (2-tailed) 


.000 

.000 


N 

936 

936 

936 

Outcome Factor 

Pearson Correlation 

.815(**) 

1 

.905(**) 


Sig. (2-tailed) 

.000 


.000 


N 

936 

936 

936 

FLSI 

Pearson Correlation 

.984(**) 

.905(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

936 

936 

936 
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FBI 2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.701(**) 

.971(**) 


Sig. (2-tailed) 


.000 

.000 


N 

305 

302 

302 

Outcome Factor 

Pearson Correlation 

.701(**) 

1 

.851(**) 


Sig. (2-tailed) 

.000 


.000 


N 

302 

309 

302 

FLSI 

Pearson Correlation 

.971(**) 

.851(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

302 

302 

302 



Behavior 

Outcome 


USMS 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.823(**) 

.985(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1471 

1471 

1471 

Outcome Factor 

Pearson Correlation 

.823(**) 

1 

.909(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1471 

1471 

1471 

FLSI 

Pearson Correlation 

.985(**) 

.909(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1471 

1471 

1471 



Behavior 

Outcome 


ATF 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.842(**) 

.988(**) 


Sig. (2-tailed) 


.000 

.000 


N 

405 

405 

405 

Outcome Factor 

Pearson Correlation 

.842(**) 

1 

.916(**) 


Sig. (2-tailed) 

.000 


.000 


N 

405 

405 

405 

FLSI 

Pearson Correlation 

.988(**) 

.916(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

405 

405 

405 


** Correlation is significant at the 0.01 level (2-tailed). 
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CIS 2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.777(**) 

.983(**) 


Sig. (2-tailed) 


.000 

.000 


N 

417 

417 

417 

Outcome Factor 

Pearson Correlation 

.777(**) 

1 

.880(**) 


Sig. (2-tailed) 

.000 


.000 


N 

417 

417 

417 

FLSI 

Pearson Correlation 

.983(**) 

.880(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

417 

417 

417 



Behavior 

Outcome 


USCG 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.796(**) 

.984(**) 


Sig. (2-tailed) 


.000 

.000 


N 

539 

539 

539 

Outcome Factor 

Pearson Correlation 

.796(**) 

1 

.892(**) 


Sig. (2-tailed) 

.000 


.000 


N 

539 

539 

539 

FLSI 

Pearson Correlation 

.984(**) 

.892(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

539 

539 

539 



Behavior 

Outcome 


USSS 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.774(**) 

.983(**) 


Sig. (2-tailed) 


.000 

.000 


N 

359 

359 

359 

Outcome Factor 

Pearson Correlation 

.11 A{**) 

1 

.878(**) 


Sig. (2-tailed) 

.000 


.000 


N 

359 

359 

359 

FLSI 

Pearson Correlation 

.983(**) 

.878(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

359 

359 

359 


** Correlation is significant at the 0.01 level (2-tailed). 
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ICE 2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.781(**) 

.981(**) 


Sig. (2-tailed) 


.000 

.000 


N 

370 

370 

370 

Outcome Factor 

Pearson Correlation 

.781(**) 

1 

.886(**) 


Sig. (2-tailed) 

.000 


.000 


N 

370 

370 

370 

FLSI 

Pearson Correlation 

.981(**) 

.886(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

370 

370 

370 



Behavior 

Outcome 


TSA 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.794(**) 

.984(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1169 

1169 

1169 

Outcome Factor 

Pearson Correlation 

.794(**) 

1 

.889(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1169 

1169 

1169 

FLSI 

Pearson Correlation 

.984(**) 

.889(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1169 

1169 

1169 



Behavior 

Outcome 


CBP 2008 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.787(**) 

.983(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1606 

1606 

1606 

Outcome Factor 

Pearson Correlation 

.787(**) 

1 

.886(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1606 

1606 

1606 

FLSI 

Pearson Correlation 

.983(**) 

.886(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1606 

1606 

1606 


** Correlation is significant at the 0.01 level (2-tailed). 
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FEMA 2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.797(**) 

.982(**) 


Sig. (2-tailed) 


.000 

.000 


N 

359 

359 

359 

Outcome Factor 

Pearson Correlation 

.797(**) 

1 

.897(**) 


Sig. (2-tailed) 

.000 


.000 


N 

359 

359 

359 

FLSI 

Pearson Correlation 

.982(**) 

.897(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

359 

359 

359 


** Correlation is significant at the 0.01 level (2-tailed). 


Table 18. 2010 Correlation Tables 


DEA 2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.843(**) 

.988(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1171 

1162 

1162 

Outcome Factor 

Pearson Correlation 

.843(**) 

1 

.914(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1162 

1181 

1162 

FLSI 

Pearson Correlation 

.988(**) 

.914(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1162 

1162 

1162 



Behavior 

Outcome 


FBI 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.781(**) 

.985(**) 


Sig. (2-tailed) 


.000 

.000 


N 

240 

234 

234 

Outcome Factor 

Pearson Correlation 

.781(**) 

1 

.877(**) 


Sig. (2-tailed) 

.000 


.000 


N 

234 

240 

234 

FLSI 

Pearson Correlation 

.985(**) 

.877(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

234 

234 

234 


** Correlation is significant at the 0.01 level (2-tailed). 
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USMS 2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.849(**) 

.989(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1641 

1628 

1628 

Outcome Factor 

Pearson Correlation 

.849(**) 

1 

.918(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1628 

1647 

1628 

FLSI 

Pearson Correlation 

.989(**) 

.918(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1628 

1628 

1628 



Behavior 

Outcome 


ATF 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.849(**) 

.989(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1653 

1644 

1644 

Outcome Factor 

Pearson Correlation 

.849(**) 

1 

.917(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1644 

1664 

1644 

FLSI 

Pearson Correlation 

.989(**) 

.917(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1644 

1644 

1644 



Behavior 

Outcome 


CIS 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.836(**) 

.989(**) 


Sig. (2-tailed) 


.000 

.000 


N 

662 

658 

658 

Outcome Factor 

Pearson Correlation 

.836(**) 

1 

.908(**) 


Sig. (2-tailed) 

.000 


.000 


N 

658 

663 

658 

FLSI 

Pearson Correlation 

.989(**) 

.908(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

658 

658 

658 


** Correlation is significant at the 0.01 level (2-tailed). 
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USCG 2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.806(**) 

.986(**) 


Sig. (2-tailed) 


.000 

.000 


N 

554 

549 

549 

Outcome Factor 

Pearson Correlation 

.806(**) 

1 

.893(**) 


Sig. (2-tailed) 

.000 


.000 


N 

549 

558 

549 

FLSI 

Pearson Correlation 

.986(**) 

.893(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

549 

549 

549 



Behavior 

Outcome 


USSS 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.810(**) 

.985(**) 


Sig. (2-tailed) 


.000 

.000 


N 

292 

287 

287 

Outcome Factor 

Pearson Correlation 

.810(**) 

1 

.898(**) 


Sig. (2-tailed) 

.000 


.000 


N 

287 

291 

287 

FLSI 

Pearson Correlation 

.985(**) 

.898(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

287 

287 

287 



Behavior 

Outcome 


ICE 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.809(**) 

.987(**) 


Sig. (2-tailed) 


.000 

.000 


N 

414 

414 

414 

Outcome Factor 

Pearson Correlation 

.809(**) 

1 

.893(**) 


Sig. (2-tailed) 

.000 


.000 


N 

414 

418 

414 

FLSI 

Pearson Correlation 

.987(**) 

.893(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

414 

414 

414 


** Correlation is significant at the 0.01 level (2-tailed). 
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TSA 2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.827(**) 

.988(**) 


Sig. (2-tailed) 


.000 

.000 


N 

1167 

1156 

1156 

Outcome Factor 

Pearson Correlation 

.827(**) 

1 

.903(**) 


Sig. (2-tailed) 

.000 


.000 


N 

1156 

1179 

1156 

FLSI 

Pearson Correlation 

.988(**) 

.903(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

1156 

1156 

1156 



Behavior 

Outcome 


CBP 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.820(**) 

.989(**) 


Sig. (2-tailed) 


.000 

.000 


N 

953 

943 

943 

Outcome Factor 

Pearson Correlation 

.820(**) 

1 

.896(**) 


Sig. (2-tailed) 

.000 


.000 


N 

943 

970 

943 

FLSI 

Pearson Correlation 

.989(**) 

.896(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

943 

943 

943 



Behavior 

Outcome 


FEMA 2010 


Factor 

Factor 

FLSI 

Behavior Factor 

Pearson Correlation 

1 

.854(**) 

.989(**) 


Sig. (2-tailed) 


.000 

.000 


N 

328 

323 

323 

Outcome Factor 

Pearson Correlation 

.854(**) 

1 

.921(**) 


Sig. (2-tailed) 

.000 


.000 


N 

323 

329 

323 

FLSI 

Pearson Correlation 

.989(**) 

.921(**) 

1 


Sig. (2-tailed) 

.000 

.000 



N 

323 

323 

323 


** Correlation is significant at the 0.01 level (2-tailed). 
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